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FOREWORD 


These  proceedings  are  the  result  of  an  invitational  conference  on  job  performance  aid 
(3PA)  cost  factors  held  in  San  Diego,  California,  2-3  3une  1982.  The  conference  was 
sponsored  by  the  Navy  Personnel  Research  and  Development  Center  and  conducted  as  part 
of  the  enlisted  personnel  individualized  career  system  (EPICS)  program. 

EPICS  research  and  development  is  guided  by  Navy  decision  coordinating  paper 
(NDCP)  Z0820-PN  (formerly  titled  Performance  Aids  Test  and  Evaluation)  and  is  under 
the  sponsorship  of  the  Deputy  Chief  of  Naval  Operations  for  Manpower,  Personnel,  and 
Training  (OP-01).  The  objectives  of  the  NDCP  are  to  define  the  state  of  the  art  in  3PA 
technology,  develop  a  conceptual  model  for  an  integrated  3PA-based  personnel  system 
including  cost  benefits  and  tradeoff  analysis,  test  the  3PA  concept,  and  quantify 
performance  increments  and  cost  benefits  obtainable  for  various  applications. 

This  report  is  the  eighth  in  a  series  of  studies  dealing  with  3PA  technology 
development:  (1)  NPRDC  TR  77-33  included  seven  papers  assessing  the  state  of  the  art  in 
3PA  technology,  (2)  NPRDC  TN  78-6  described  a  preliminary  enlisted  personnel  system 
concept  with  major  emphasis  on  the  use  of  3PAs,  (3)  NPRDC  TR  78-26  was  a  systematic 
review  and  organization  of  existing  3PA  techniques,  related  research  data,  and  various 
applicable  principles  and  concepts,  (4)  NPRDC  TN  79-1  defined  a  3PA  selection  algorithm 
or  an  integrated  personnel  system,  (5)  NPRDC  TR  79-25  discussed  development  of  hybrid 
and  enriched  hybrid  troubleshooting  3PAs,  (6)  NPRDC  TR  82-7  described  the  development 
and  test  of  a  troubleshooting  aid  for  digital  systems,  and  (7)  NPRDC  SR  83-32  described  a 
field  evaluation  of  enriched  hybrid  troubleshooting  3PAs.  The  purpose  of  the  conference 
was  to  bring  together  3PA  researchers  and  developers  to  define  the  factors  influencing 
3PA  cost,  the  measurement  and  estimation  of  these  factors,  the  weight  of  their  influence, 
and  the  identification  of  potential  guidelines  for  accurate  cost  estimation. 

The  presentations  are  essentially  a  verbatim  transcript  of  each  individual's  remarks. 
Each  presentation  is  followed  by  a  discussion  section  that  summarizes  the  comments  by 
all  participants  during  and  after  the  individual's  presentation.  Abbreviations  and 
acronyms  are  used  throughout.  Therefore,  the  attention  of  the  reader  is  invited  to  the  list 
of  abbreviations  and  acronyms  that  follows  the  references. 


3AMES  F.  KELLY,  3R. 
Commanding  Officer 


3AMES  W.  TWEEDDALE 
Technical  Director 
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SUMMARY 


Background  and  Purpose 


An  invitational  conference  sponsored  by  the  Navy  Personnel  Research  and 
Development  Center  (NAVPERSRANDCEN)  was  held  in  San  Diego,  California  during  2-3 
June  1982.  The  purpose  of  the  conference  was  to  identify  factors  that  influence  job 
performance  aid  (JPA)  cost  and  to  discuss  the  measurement  and  prediction  of  these 
factors.  Individuals  were  invited  from  three  government  and  eight  industry  locations. 
Individual  presentations  were  intermixed  with  group  discussion  and  interaction.  This 
report  provides  the  content  of  the  presentations,  a  summary  of  the  discussion  surrounding 
each  presentation,  and  a  summary  of  the  cost  factors  and  conclusions  regarding  cost 
prediction. 

Introduction 


The  meeting  was  opened  by  Dr.  Glenn  Osga  of  Systems  Exploration,  Incorporated, 
who  welcomed  everyone,  and  Dr.  Robert  Blanchard  of  NAVPERSRANDCEN,  who  briefly 
discussed  the  context  of  the  meeting  as  generating  from  the  continuing  enlisted  personnel 
individualized  career  system  (EPICS)  research  effort.  He  identified  the  existence  of 
inadequacies  in  cost  estimation  and  the  importance  of  identifying  cost  within  a  cost- 
versus-effectiveness  framework  for  different  technical  data  presentation  methods.  Dr. 
Blanchard  emphasized  the  use  of  an  integrated  personnel  systems  approach,  suggesting 
that  JPAs  not  be  emphasized  to  the  exclusion  of  other  methods/considerations  with 
overall  cost  impacts  on  personnel  career  development.  Dr.  Robert  Smillie  of  NAVPERS¬ 
RANDCEN  followed  Dr.  Blanchard's  comments  with  some  general  comments  that  served 
to  focus  the  meetings.  Dr.  Smillie  proposed  a  question  for  the  meeting:  Looking  from  the 
project  director's  point  of  view  with  user  skill/job  level  and  budget  defined  for  technical 
data  procurement,  what  are  the  cost  factors  and  what  are  the  cost  effectiveness 
tradeoffs? 

Presentations 


In  the  first  presentation,  Mr.  Don  Finegan  of  the  U.S.  Army  presented  an  overview  of 
a  cost  and  volume  study  being  conducted  by  the  Army  for  the  purpose  of  improving  skilled 
performance  aid  (SPA)  specifications.  Mr.  Finegan  outlined  a  number  of  cost  drivers, 
including  level  of  detail,  illustration  quantity,  color  vs.  black  and  white,  logistics  support 
analysis  requirements  (LSAR)  data  use,  and  cost  impacts  of  numerous  presentation 
techniques  and  technical  data  development  procedures.  He  concluded  with  suggestions  for 
cost  reduction  such  as  using  training  people  to  write  technical  manuals  and  improving  JPA 
specifications.  During  discussion,  it  was  determined  that  procurement  people  must  be 
trained  to  understand  the  flexibility  contained  in  specifications  and  what  is  required  by 
the  government.  Allowing  a  certain  latitude  in  specifications  is  all  right  if  the  procurers 
are  knowlegeable  and  effective. 

Mr.  Sam  Rainey  of  the  David  Taylor  Naval  Ship  Research  and  Development  Center 
outlined  major  cost  drivers  from  the  Navy's  perspective.  He  focused  on  the  effectiveness 
of  the  system  acquisition  manager  as  a  major  cost  determinant.  Mr.  Rainey  pointed  to 
inadequate  input  data  in  the  form  of  poor  logistics  support  analysis  (LSA)  and  indicated 
that  much  technical  material  is  done  "just  to  get  it  in  on  time"  with  poor  quality  control. 
In  the  cost  vs.  quality  decision,  however,  lower  cost  usually  wins  because  upper 
management  people  do  not  recognize  quality  but  do  know  when  budgets  are  overrun.  Mr. 
Rainey  concluded,  however,  that  cost  must  be  viewed  as  secondary  to  effectiveness  and 
never  in  isolation. 
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Mr.  Ted  Post  of  BioTechnology,  Inc.  discussed  present  methods  of  3PA  production 
through  the  use  of  a  master  book  plan  from  which  a  task  identification  matrix  is  formed. 
This  leads  to  cost-per-page  information  based  on  labor  rates,  pages  of  illustration,  etc. 
Mr.  Post  explained  that,  all  too  often,  knowledge  of  this  cost-estimation  process  resides 
with  contractors,  not  contract  monitors.  Also,  Mr.  Post  suggested  that  better  use  of 
historical  data  be  implemented,  such  as  done  by  Project  Hardman,  to  determine  technical 
data  needs  and  costs  before  the  design  freeze  stage.  He  stated  that  the  user-data  match 
is  important  and  that  3 PA  developers  should  be  involved  early  in  the  procurement  process 
to  participate  in  tradeoff  decisions. 

Mr.  Fred  Hart  of  Kinton,  Inc.  discussed  3PA  cost  factors  for  the  development  of 
procedural  and  functional  JPAs.  Procedural  3PAs  were  discussed  within  a  six-stage 
development  process;  and  functional  3PAs,  within  a  variable  stage  process  dependent  upon 
the  type  of  illustration/text  used.  Specific  cost  drivers  were  presented  for  each  process 
step.  Mr.  Hart  concluded  that  the  procuring  agency  must  know  what  the  technical 
information  needs  are  to  complement  a  given  system,  and  this  information  should  not 
simply  be  a  volume  of  data  not  addressing  user  needs.  Lack  of  needs  analysis  during  early 
development  of  a  system  was  cited  as  a  major  problem  contributing  to  procuring  agencies' 
inefficiency. 

Dr.  Kay  Inaba  of  Xyzyx  Information  Company  placed  cost  considerations  as  the 
biggest  contributing  factor  inhibiting  the  implementation  of  3PAs  in  the  field  because 
problems  (and  cost  uncertainty)  are  anticipated  in  view  of  unsure  production  specifica¬ 
tions.  Dr.  Inaba  made  a  distinction  between  "front-end"  analysis  work  and  3PA  generation 
work  during  his  presentation.  He  felt  that  his  company  has  been  able  to  stabilize 
generation  costs  with  computer-aided  authoring.  Major  cost  factors  discussed  were  the 
quality  of  background  technical  data  available  as  input,  the  inclusion/exclusion  of 
troubleshooting  tasks,  and  the  production  time  necessary  for  translating  ideas  into 
sentences.  Dr.  Inaba  discussed  a  computer  program  to  aid  sentence  writing,  his 
techniques  for  dealing  with  background  data  quality,  and,  finally,  possible  ways  of 
reducing  cost  for  troubleshooting  tasks.  Computerized  text  processing  and  storage  were 
also  suggested  as  cost  reducers. 

Mr.  William  Conroy  of  the  Raytheon  Service  Company  presented  an  example  of  the 
development  of  maintenance  requirement  cards  (MRCs)  with  an  estimate  of  man-hours  for 
a  given  development  project  based  upon  local  equipment  availability  and  other  specified 
personnel/background  data  states.  He  then  compared  actual  hours  used  and  presented 
specific  reasons  for  prediction  errors.  Mr.  Conroy  questioned  the  overkill  in  manual 
procurement  and  the  need  for  verification/validation  using  100  percent  of  the  procedures. 
He  suggested  that  level  of  detail,  amount  of  artwork,  personnel  safety  required,  and 
working  environment  factors  combine  to  influence  cost  substantially  by  dictating 
quality/type  of  technical  data  necessary  for  the  on-the-job  user. 

Ms.  Rosemarie  Preidis  of  the  Air  Force  Human  Resources  Laboratory  (AFHRL), 
presented  data  from  an  ongoing  project  that  has  resulted  in  the  development  of  a 
computer  algorithm  designed  to  predict  page  quantity  and  type  for  technical  manuals, 
given  the  number  of  subsystems,  line  replaceable  units  (LRUs),  and  shop  replaceable  units 
(SRUs).  A  regression  equation  that  estimates  12  types  of  page  quantities,  given  these 
inputs,  was  developed.  She  presented  an  example  in  which  cost  data  from  three 
contractor  sources  were  inserted  into  the  model  to  estimate  total  production  costs. 

Mr.  Fran  Rahl  of  Westinghouse  Electric  Corporation  described  cost  factors  as  either 
3PA  attributes  or  circumstances  surrounding  3PA  development.  3PA  attributes  described 
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were  text/graphics  mix,  data  density,  and  gross  page  volume.  Development  circum¬ 
stances,  such  as  customer  type,  competition,  exist ence/availability  of  hardware,  etc., 
were  described  as  difficult  to  quantify,  not  apparent  in  the  final  product  appearance,  and 
influencing  cost  more  than  3PA  characteristics.  Mr.  Rahl  concluded  that  price  models 
would  be  of  most  use  in  performing  tradeoff  analysis,  as  opposed  to  estimating  final  cost. 

Mr.  3ohn  Weber  of  the  Lockheed-California  Company  presented  contractor  staff 
characteristics,  project  schedule,  and  overhead  costs  as  primary  factors  affecting  the 
quality  and  cost  of  3PA  production.  Mr.  Weber  explained  how  the  request  for  quotation 
(RFQ)  response-time  schedule  can  undermine  the  competitive  bidding  system  and  affect 
quality  and  cost  of  the  final  product.  Staffing  problems  in  3PA  production  are  difficult  to 
control  as  a  result  of  lack  of  continuity  for  3PA  work  and  stringent  training  requirements 
for  3PA  writers.  Company  overhead  rates  were  also  seen  as  a  major  cost  driver. 
Mr.  Weber  concluded  that  the  labor-intensive  3PA  development  process  is  most  directly 
affected  by  the  contractor  in-house  personnel  system  attributes  and,  to  a  lesser  degree, 
by  3PA  attributes  and  formats. 

Mr.  Reid  3oyce  of  Applied  Science  Associates  reviewed  conference  presentations, 
discussed  the  "system  acquisition  manager  problem"  and  emphasized  the  need  to  institu¬ 
tionalize  changes  in  the  procurement  process  related  to  3PA  acquisition.  Mr.  3oyce 
presented  a  discussion  of  new  positions  that  would  be  created  as  part  of  this  institutional 
change.  He  feels  that  there  are  so  many  ways  of  getting  around  poor  cost  estimates  that 
the  motivation  for  accurate  estimates  must  be  legislated  and  driven  by  such  institutional 
changes. 

Mr.  3ohn  Bean  of  the  Hughes  Aircraft  Company  compared  two  methods  of  cost 
estimation.  The  first  estimated  total  cost  as  a  function  of  historical  similarity  between 
new  and  previous  systems  combined  with  some  function  of  the  new  system  (like  number  of 
LRUs,  SRUs,  etc.).  The  second  method  used  work  elements  such  as  writing,  editing, 
photography,  etc.  combined  with  the  system  functions.  Mr.  Bean  discussed  the  accuracy 
of  each  method,  the  key  cost  elements,  and  problems  associated  with  utilizing  "cost-per- 
page"  in  estimating  project  costs.  He  offered  recommendations  such  as  direct  pickup  of 
technical  information  source  data,  making  an  effort  to  keep  technical  information  to  an 
essential  minimum,  and  inclusion  of  work  samples  as  part  of  the  request  for  proposal 
(RFP). 

Dr.  Glenn  Osga  of  Systems  Exploration,  Incorporated  summarized  the  presentations 
of  the  11  speakers  and  grouped  the  salient  points  as  cost  factors,  cost  reduction 
suggestions,  and  cost  estimation  techniques.  Dr.  Osga  presented  these  topics  within  a 
3PA  development  framework  that  comprises  seven  areas:  (1)  personnel,  equipment,  and 
environment  characteristics,  (2)  procuring  agency  attributes,  (3)  3PA  producer  attributes, 
(4)  3PA  characteristics,  (5)  RFP/bidding  processes,  (6)  input  data  quality,  and  (7)  3PA 
production  process.  For  each  of  these  areas,  the  most  important  cost  drivers  are 
identified.  Existing  quantitative  models  need  to  be  validated  and  expanded  to  include  the 
myriad  phases  during  3PA  production  with  a  separate  focus  on  the  front-end  cost  analysis. 
The  front-end  3PA  development  procedures  and  methods  are  so  variable  that  accurate 
estimations  of  front-end  costs  cannot  be  made  until  responsibility  for  interim  products  is 
established.  By  reducing  the  variability  in  front-end  preparation  processes,  historical  cost 
data  can  be  used  to  develop  models  such  as  the  current  AFHRL  effort. 


IX 


Recommendations 


1.  Identify  areas  where  built-in  test  equipment  (BITE)  and  automated  test  equip¬ 
ment  (ATE)  may  be  harmful  to  the  troubleshooting  technician  as  well  as  the  cost  tradeoffs 
for  BITE  and  ATE. 

2.  Identify  areas  in  the  front-end  analysis  (e.g.,  the  failure  mode  effects  analysis 
and  dependency  analysis)  that  do  not  provide  adequate  troubleshooting  data  for  develop¬ 
ment  of  troubleshooting  3PAs. 

3.  Develop  guidelines  for  3PA  procurers  that  account  for  symmetry  in  electronic 
equipment. 

4.  Improve  cost/effectiveness  tradeoff  guidelines  for  the  user/data  match. 

5.  Improve  the  methodology  for  quantifying  the  3PA  cost  impact  elements. 

6.  Implement  an  education  program  for  acquisition  managers  and  procurement 
personnel  that  demonstrates  the  long-term  life  cycle  cost  effectiveness  of  the  3PA 
methodology. 

7.  Develop  criteria  to  reduce  the  variability  of  the  attributes  of  procuring  agencies. 

8.  Develop  strategy  that  places  the  burden  on  the  contractor  for  an  accurate  cost 
estimate. 

9.  Develop  guidelines  for  including  work  samples  in  procurement  packages. 

10.  Improve  state-of-the-art  methods  for  defining  and  quantifying  cost-format- 
performance  relationships. 

11.  Develop  guidelines  for  establishing  a  bidders'  conference  that  can  be  used  to 
increase  the  understanding  of  requirements  and  clarify  customer  uncertainty. 

12.  Develop  guidelines  that  adequately  reflect  the  separation  of  front-end  costs 
from  3PA  production  costs. 

13.  Develop  guidelines  that  delineate  the  responsibilities  for  quality  input  data  for 
the  3PA  development  process. 

14.  Determine  the  relationship  of  LSA  to  the  3PA  development  process  and  identify 
the  3PA  input  data  gaps. 

15.  Establish  cost-effective  guidelines  for  3PA  validation/verification  that  address 
task  sample  size,  user  sample  size,  and  availability  of  equipment. 

16.  Identify  specific  areas  where  production  specifications  are  forcing  increased 
volume  and  cost,  and  develop  guidelines  for  improving  the  process. 
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INTRODUCTION 


Dr.  Glenn  A.  Osga 
Systems  Exploration,  Incorporated 
San  Diego,  California 


Background 

In  an  introduction  to  a  previous  conference  on  the  status  of  job  performance  aid 
(JPA)  technology,  Blanchard  (1977)  noted  the  existance  of  piece-meal  implementation  of 
JPAs  by  the  military  services  despite  numerous  studies  suggesting  positive  reductions  in 
training  and  maintenance  costs.  During  the  present  conference,  it  was  suggested  that 
cost  was  the  biggest  single  barrier  impeding  JPA  implementation.  When  JPAs  are  thought 
of— high  cost  must  follow.  The  validity  of  this  statement  may  or  may  not  be  true,  given 
that  other  methods  of  technical  documentation  are  not  inexpensive.  Considering  cost 
without  considering  effectiveness  should  not  be  done,  at  least  in  theory.  We  are  not  able 
to  obtain  very  good  measurements  of  effectiveness  in  performance  terms,  given  the  state 
of  our  methods  and  the  real-world  restrictions.  Thus,  cost  is  quite  often  a  solitary 
consideration. 

First,  consider  the  estimation  of  cost  for  a  technical  data  producing  project.  How  is 
it  done  and  what  motivates  the  preparer  to  be  accurate?  Are  high  estimates  due  to  fear 
of  problems  and  unsure  customer  guidance?  Are  estimates  better  based  on  historical  data 
or  careful  addition  of  cost  elements  for  a  current  project? 

Second,  considering  the  high  cost  of  technical  data  production,  how  can  we  reduce 
overall  costs  and,  therefore,  increase  the  likelihood  of  implementation?  Can  the 
customer  decide  what  technical  information  format  is  most  cost  effective  for  his  system 
or  does  he  care?  If  he  does  care,  how  can  the  JPA  community  guide  his  decisions  through 
carefully  planned  proposal  and  contract  meetings?  How  does  the  customer  compare 
proposals  and  what  are  the  cost  effects  of  poor  background  data  collection  or  specifica¬ 
tion  implementation?  These  varied  questions  surrounding  the  topic  of  JPA  preparation 
cost  provided  the  impetus  to  conduct  this  work  group  meeting. 

Purpose 

Consideration  of  these  cost  questions  stems  from  ongoing  efforts  at  the  Navy 
Personnel  Research  and  Development  Center  (NAVPERSRANDCEN)  to  increase  the 
understanding  of  costing  problems  in  JPA  systems  and,  therefore,  advance  the  state  of  the 
art  with  respect  to  their  implementation.  The  information  collected  during  these 
discussions  could  then  provide  a  groundwork  for  attacking  the  cost  problem  at  different 
fronts. 

The  thought  behind  the  present  conference  was  to  provide  an  informal  atmosphere 
where  a  group  of  government  and  contractor  JPA  specialists  could  voice  their  opinions 
about  factors,  situations,  necessities,  etc.  that  affect  cost  and  cost  estimation.  First,  it 
was  hoped  that  cost-affecting  processes,  regulations,  specifications,  key  personnel, 
methodologies,  and  various  facts  of  life  found  within  both  government  and  industry 
settings  would  be  identified.  Second,  it  was  anticipated  that  the  government  would 
present  desires  (or  demands)  to  the  contractors  and  vice  versa.  Third,  and  most 
important,  it  was  hoped  that  information  exchange  would  not  stop  with  identification  and 
statement  of  various  desires,  but  that  concrete  recommendations  for  improving  these 
problems  might  be  voiced. 
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Approach 


An  informal  approach  was  utilized  in  which  participants  were  asked  to  prepare  a 
short  briefing  on  their  experience  in  cost  estimation  and  how  they  felt  it  could  be  better 
done  with  costs  better  controlled.  Participants  used  notes  but  talked  freely. 

These  proceedings  present  a  synopsis  of  the  oral  and  written  information  submitted  at 
the  2-day  meeting.  Presentations  varied  greatly  in  the  relative  amount  of  discussion  and 
presentation.  Each  participant's  material  is  presented,  followed  by  a  synopsis  of  the 
discussion  during/after  the  presentation.  Synposis  of  discussion/presentations  were 
prepared  from  various  sources  including  conference  notes,  secretarial  notes,  taped 
conference  proceedings,  proceedings,  and  written  materials  submitted  by  presenters. 
Hopefully,  they  will  provide  guidance  for  further  study  of  cost  problems. 
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OPENING  COMMENTS 


Dr.  Robert  E.  Blanchard  and  Dr.  Robert  3.  Smillie 
Navy  Personnel  Research  and  Development  Center 
San  Diego,  California 


Dr.  Blanchard 


I  would  like  to  discuss  the  background  behind  this  conference.  Most  of  you  are  aware 
of  the  EPICS  program,  which  has  been  conducted  at  NAVPERSRANDCEN  during  the 
previous  6  years.  At  the  present  time,  we  have  developed  what  we  feel  are  state-of-the- 
art  3PAs,  which  have  been  implemented  on  34  Navy  ships  over  the  past  year  and  a  half. 
An  important  part  of  the  3PA  methodology  and  of  the  EPICS  program  is  development  of 
methodology  that  can  be  used  in  future  procurements.  We  hope  to  develop  and  extend  the 
methodology  that  has  evolved  during  the  EPICS  program  and  implement  this  in  training, 
3PA,  and  job  design  projects,  following  thorough  field  testing.  One  of  the  things  that 
we've  discovered  in  trying  to  model  this  complex  personnel  system  is  that  we're  really  not 
very  adept  at  anticipating,  forecasting,  or  modeling  cost  and  incorporating  cost  tradeoffs 
with  the  other  personnel  factors.  For  example,  a  very  straightforward  type  of  tradeoff 
that  we  have  considered  is  that  between  3PAs  and  training  utilization.  Although  the  use 
of  3PAs  over  training  is  really  the  "battle  cry"  of  the  EPICS  program,  what  we  have  done 
in  fact  is  replace  very  expensive  training  programs  with  very  expensive  3PAs.  One  must 
consider,  however,  that  trained  personnel  matriculate  through  the  system  and  3PAs  are 
with  us  to  stay. 

What  we  need,  ladies  and  gentlemen,  is  some  help  in  how  to  evaluate  costs  and  cost 
modeling,  and  how  these  factors  are  related.  I  guess  the  only  comment  that  I  would  offer 
is  that  you  not  get  buried  in  3PA  costing  to  the  exclusion  of  other  factors.  At 
NAVPERSRANDCEN,  we  feel  very  strongly  about  integrated  personnel  system  approaches 
to  these  problems.  In  other  words,  try  not  to  get  your  eyes  "too  close  to  the  table,"  as  it 
may  be,  in  considering  just  technical  data.  You  will  be  concerned  with  these  things  during 
the  course  of  this  meeting.  However,  we  are  really  interested  in  overall  cost  and  cost 
impacts  in  things  of  this  nature.  Appreciate  that  we  are  after  a  model— something  that 
we  can  implement  in  the  personnel  system  and  use  to  offer  alternatives,  options,  tradeoffs 
in  costing  out  these  factors.  Costs  are  extremely  important  in  the  world  we  deal  with 
today.  I'd  like  to  wish  you  the  best  of  luck  during  this  conference  and  I  hope  the  dialogue 
runs  freely  and  that  we  really  accomplish  something. 

Dr.  Smillie 


What  we're  really  trying  to  get  and  to  come  up  with  at  the  end  of  this  conference  is  a 
general  direction,  as  opposed  to  definitive  conclusions.  I  don't  think  we  will  be  able  to  get 
everything  laid  out  as  far  as  what  the  cost  factors  are  and  how  they  interact.  Assume, 
from  a  project  director's  point  of  view,  that  you  are  procuring  technical  documentation 
that  has  to  accompany  a  specific  system.  Therefore,  what  kind  of  guidance  can  we  give 
such  a  person  when  he  knows  (1)  what  his  user  audience  is  going  to  be,  and  (2)  what  his 
overall  budget  is  for  buying  technical  data?  How  does  he  decide  what  type  of  format  to 
utilize  for  the  3PA  or  what  proportion  of  his  dollars  to  sink  into  training  vs.  paper-type 
3PA  products?  We're  not  trying  to  get  into  specifically  what  Lockheed  or  Xyzyx  charge 
when  they  are  developing  job  performance  aids.  What  we  really  are  after  is  what 
components  they  consider  when  they  make  a  bid  on  such  a  project.  They  know,  for 
example,  that  a  certain  amount  of  illustrating  hours  will  be  necessary  for  the  pictorial 
portions  of  the  3PA  and  that  these  will  be  costly.  With  these  comments  in,  we  can  get 
started  on  our  first  presentation. 
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IMPROVING  THE  SPECIFICATIONS  FOR  SKILLED  PERFORMANCE  AIDS 


Mr.  Don  Finegan 
U.S.  Army  DARCOM-MERSA 
Lexington,  Kentucky 


Presentation 


I  have  brought  an  outline  of  a  cost  and  volume  study  which  the  Army  is  currently 
conducting.  The  Army  buys  what  are  called  "skilled  performance  aids,"  or  SPAs,  that  are 
similar  to  job  performance  aids.  I  will  present  two  slides  concerned  with  this  study. 
Specifically,  this  study  covers  two  specifications  that  the  Army  uses  to  procure  SPAs. 

Slide  1  shows  basically  the  time  frame  that  we  are  looking  at.  We  are  at  this  point  in 
time  in  the  current  project  (refer  to  Item  25  on  slide  1),  and  part  of  what  I  came  here  for 
is  to  obtain  contractor  input  on  what  your  cost  drivers  are.  Also,  what  volume  problems 
you  have  had  in  your  experiences.  Our  experience  has  been  that  the  specifications  are 
often  misinterpreted  due  to  their  great  latitude  with  the  effect  of  volume  going  up;  we'd 
like  to  get  some  control  over  that  situation. 


Finegan:  Slide  1 
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What  we  are  going  to  do  is  to  put  together  a  package,  coordinate  with  TRADOC,  go 
out  to  the  various  commands,  and  brief  them  on  this  project.  The  packages  that  we 
present  to  them  will  probably  contain  elements  which  we  will  include  in  future  contracts, 
until  we  can  make  the  changes  in  the  specifications.  Since  it  will  probably  be  some  time 
until  the  specifications  are  changed,  we  hope  that  this  package  will  serve  as  an  interim 
guide  for  our  contract  monitors.  Slide  2  shows  the  specific  areas  that  we  are  investiga¬ 
ting.  An  example  of  overkill  would  be  six  pages  of  text  to  inflate  a  tire.  We  are 
investigating  the  use  of  color  in  things  like  dependency  charts  and  wiring  diagrams. 


AREAS  BEING  INVESTIGATED 


OVERKILL,  i.E..  TOO  MUCH  DETAIL  FOR  SIMPLE  PROCEDURES. 

QUANTITY  OF  ILLUSTRATIONS  -  COST  BENEFIT  VERSUS  INCREASE  IN  USABILITY  FACTORS. 

GUIDELINES  FOR  USE  OF  COLOR  AS  RELATED  TO  COST/USABILITY  FACTORS. 

USE  OF  LOGISTICS  SUPPORT  ANALYSIS  RECORD  (LSAR)  DATA  -  COST  IMPACT  UPON  PUBLICATIONS. 
INSTRUCTIONS  FOR  TAILORING  SPECIFICATIONS. 

VALIDATION  AND  VERIFICATION  COSTS  (METHODS  AND  COSTS). 

SPECIFICATION  REQUIREMENTS.  ESPECIALLY  THOSE  CITING  REQUIREMENTS  FOR  DETAIL  STEP-BY-STEP 
PROCEDURES  (REF.  PAGES  6/2,  6/9,  6/10,  7/1.  7/2.  AND  7/3  OF  M I L-HDBK-63038-1 . 

USE  OF  REFERENCES  (CLARIFICATION). 

USE  OF  LOCATOR  VIEWS  AND  HUMAN  FIGURES. 

USE  OF  CURTAILED  TEXT. 

COVERAGE  FOR  SYMMETRICALLY  OPPOSITE  HARDWARE  AND  LIKE  ITEMS. 

COVERAGE  OF  MAINTENANCE  PROCEDURES  AS  LOGICAL  TASKS/JOBS  RATHER  THAN  PIECEMEAL  TYPE 
COVERAGE  OF  EVERY  ITEM  LISTED  ON  THE  MAC. 

USE  OF  TABULAR  PRESENTATION  TECHNIQUES. 


Finegan:  Slide  2 


The  Army  uses  target  audience  personnel  during  their  validation  of  3PA  materials  and 
this  tends  to  increase  cost.  Regarding  the  use  of  references,  when  we  went  to  integrated 
art  and  text,  we  did  away  with  the  reference  of  illustrations  in  the  text.  We  do  allow 
some  limited  use  of  illustration  references  in  cases  where  the  equipment  is  "torn  down" 
prior  to  maintenance.  But  we  have  found  in  many  cases  that  this  is  not  utilized  and 
procedures  are  described  repeatedly  throughout  the  text. 

Regarding  the  use  of  locator  views  and  human  figures,  we  want  to  restrict  the  use  of 
locator  views  to  one  per  procedure.  The  use  of  curtailed  text  depends,  of  course,  on  the 
target  audience.  Basically,  we  hope  to  use  illustrations,  where  possible,  to  show  items, 
rather  than  lengthy  textual  descriptions.  To  sum  up,  these  are  the  main  areas  that  we  are 
investigating  right  now.  Any  kind  of  input  we  can  get  from  the  contractor's  side 
concerning  specification  improvements  or  factors  that  we  haven't  even  considered  would 
be  appreciated. 
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Discussion 


Initial  discussion  centered  on  the  use  of  color  in  SPAs.  Early  specifications  pushed 
the  use  of  color  in  some  of  the  procedural  SPAs.  Mr.  Finegan  said  that  a  survey  was  under 
way  to  obtain  subjective  data  on  user  opinion  of  color.  Color  has  been  shown  to  influence 
some  types  of  information  transfer  positively  and  have  no  effect  on  others.  Color  use  on 
wiring  diagrams  was  discussed  and  opinions  were  mixed.  Gray  shading  was  suggested  as  a 
lower-cost  (but  still  expensive)  alternative.  Color  is  often  used  at  the  whim  of  the 
individual  buyer  regardless  of  the  specifications  regulating  the  use  of  color. 

Discussion  then  focused  on  specifications  and  in-process  reviews.  Lack  of  start-of- 
work  meetings  and  improper  in-process  reviews  were  cited  as  significant  cost  drivers. 
These  events  are  hampered  by  poor  government  specifications  of  the  products  it  wants. 
The  need  for  thorough  government  analysis  and  review  of  systems  needs  prior  to  the 
contractor  involvement  was  emphasized.  Mr.  Finegan  stated  that  this  was  the  purpose  of 
the  "package"  they  hoped  to  create — to  provide  a  checklist  with  examples  of  items  that 
they  (the  customer)  could  utilize.  Also,  a  need  was  voiced  for  follow-up  on  written 
specifications  to  ensure  they  are  followed  and  understood  by  procurement  specialists.  The 
need  for  training  to  accompany  the  specifications  was  emphasized  by  participants. 
Benefit  of  latitude  in  specifications  was  discussed  and  the  latitude  is  reduced  when 
requirements  are  tailored  to  the  individual  project.  It  was  agreed  that  this  latitude  is  of 
benefit  only  when  contact  procurers  were  knowledgeable  and  effective. 

Mr.  Finegan  mentioned  the  possible  specification  of  a  tabular  format  for  maintenance 
procedures  to  restrict  volume.  Discussion  of  specifications  ended  with  a  brief  dialogue 
concerning  modified  specifications  and  contractor  training  of  writers  for  implementing 
specifications  in  their  writing.  Modified  specifications  were  mentioned  as  easing  writing 
jobs  by  reducing  the  redundancy  of  text;  however,  it  was  noted  that  these  specifications 
also  can  overpower  the  writer  with  a  large  volume  of  information. 

The  use  of  logistic  support  analysis  record  (LSAR)  data  and  the  effect  of  use  on  cost 
was  discussed.  It  was  agreed  that  LSAR  could  be  useful,  but  often  was  not,  because  it  was 
not  properly  collected.  Part  of  the  problem  lies  in  the  endeavor  of  LSAR  data  to  be 
everything  to  many  different  disciplines,  while  not  specifically  for  technical  manual 
developers. 

Discussion  of  validation/verification  was  controversial  concerning  the  effectivenss  of 
these  techniques  and  appropriate  methodology  for  conducting  them,  and  the  feasibility 
and  cost  of  utilizing  100  percent  of  the  tasks  involved.  Target  audience  use  was  agreed  to 
be  necessary,  but  other  topics  were  unresolved.  Verification  of  troubleshooting  proce¬ 
dures  was  discussed.  The  use  of  subject  matter  expert  input  for  technical  accuracy  and 
target  audience  input  for  comprehensibility  was  suggested.  Group  consensus  was  that 
fault  insertion  was  the  generally  used  method  of  verifying  troubleshooting;  however,  the 
method  of  selection/sampling  of  faults  varied  and  was  dependent  on  project  management. 

Discussion  of  task  overlap  centered  on  the  problem  of  analyzing  tasks  and  eliminating 
pages  of  overlapped  information  common  to  many  different  tasks.  The  group  agreed  that 
this  analysis  should  be  done  early  and  that  the  contractor  will  generally  not  have  the  time 
or  money  to  do  this.  A  suggestion  was  made  that  an  initial  front-end  analysis  be  done  by 
the  government  to  serve  as  a  guideline  for  the  project  monitoring  process.  The  use  of  the 
training  people  as  writers  could  reduce  task  analysis  and  data  overlap  problems,  but  this 
would  be  more  expensive  than  using  conventional  technical  writers. 
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MAJOR  TECHNICAL  INFORMATION  COST  DRIVERS 
FROM  THE  NAVY'S  PERSPECTIVE 

'  V  * 

Mr.  Samuel  C.  Rainey 

David  Taylor  Naval  Ship  Research  and  Development  Center 
Bethesda,  Maryland  20084 


Presentation 


I  said  earlier  that  one  of  the  biggest  factors  that  affects  technical  information  (TI) 
cost,  at  least  in  the  Navy,  stems  from  the  systems  acquisition  managers  themselves  and, 
in  particular,  those  in  the  Naval  Sea  Systems  Command — which  buys  90  percent  of  the 
Navy's  documentation.  An  acquisition  manager  might  spend  large  sums  of  money  for  TI— 
60  million  dollars,  for  example — and  be  primarily  concerned  that  he  get  the  material 
delivered  on  time  without  equivalent  concern  for  the  quality  of  the  material.  It  takes  a 
lot  of  time  just  trying  to  get  the  attention  of  people  like  that  and  they  are  buying 
approximately  3000  manuals  per  year  (with  at  least  some  form  of  TI  for  100,000  pieces  of 
equipment  per  year).  In  general,  no  one  reviews  those  manuals — the  quality  control  is 
negligible.  So  we  sit  at  meetings  like  this  trying  to  come  up  with  solutions  and  suggest 
improvements  in  the  TI  generation  processes,  when  the  problem  really  seems  to  be 
implementation  of  these  solutions  once  they  are  suggested. 

How  important  is  the  TI  cost  in  relation  to  consideration  of  TI  effectiveness  anyway? 
If  I  spend  $1,000  per  page  and  the  equipment  works  and  has  less  downtime,  then  that 
money  is  well  spent.  If  I  spend  50  percent  of  that  amount  and  the  TI  is  inadequate,  I  must 
have  it  done  over  again  at  higher  cost.  If  I  lost  my  equipment  to  downtime  for  a  third  of 
the  time,  then  I've  lost  far  more  money  in  comparison  (to  the  TI  cost).  Really,  the  only 
cost  of  major  interest  is  the  cost  of  system  ownership.  By  that  I  mean  dollars  per  hour 
that  the  equipment  works. 

My  point  is  that  system  acquisition  managers  are  not  rewarded  for  buying  good 
technical  information.  They  are  rewarded  when  they  complete  the  purchase  of  a  certain 
number  of  airplanes  or  hardware  systems,  regardless  of  the  quality  of  the  TI  and  other 
logistic  support  measures. 

Also,  inadequate  integrated  logistics  support  (ILS)  and  logistic  support  analysis  (LSA) 
contribute  to  higher  TI  costs  because  the  technical-manual  generator  has  to  repeat  the 
analysis  not  done  effectively  in  the  first  place  before  he  can  begin  his  job. 

In  the  Navy  technical  information  presentation  program  (NTIPP),  we  have  often  found 
that  the  engineering  data  base  is  formulated  in  such  a  way  that,  when  it's  handed  over  to 
the  technical  writers,  it  has  to  be  extensively  reworked.  This  is  costly.  Another  cost 
driver  is  the  need  to  implement  certain  comprehensibility  approaches  in  the  Navy  under 
NTIPP.  However,  NTIPP-developed  comprehensibility  approaches  will  be  accompanied  by 
a  good  deal  of  automation,  which  will  help  keep  the  cost  from  rising  further. 

Discussion 


Discussion  of  TI  cost  vs.  effectiveness  centered  on  participant  disagreement  as  to  the 
relative  merit  of  the  two  factors.  It  was  agreed  that  most  often  the  persons  who  can 
make  tradeoff  decisions  regarding  cost  vs.  quality  will  make  decisions  that  decrease  cost, 
but  produce  a  sacrifice  in  quality,  because  these  top-level  people  do  not  inspect  the 
products  and  would  not  know  quality  if  they  saw  it.  They  do  know  when  budgets  are 
overrun,  however. 
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Mr.  Rainey  identified  two  additional  factors  that  he  considered  as  contributing  to 
increased  TI  cost: 

1.  The  failure  to  integrate  the  needs  of  the  training  community  into  the  initial  TI 
preparation  process,  thus  requiring  that  additional  technical  data  be  developed  for  use  in 
the  schools. 

2.  The  incorporation  of  material  into  TI  that  cannot  be  used  by  the  technicians,  but 
which  satisfies  specifications. 

The  problem  of  updating  TI  when  design  changes  are  made  was  discussed.  It  was 
agreed  that  changes  are  often  done  informally  by  an  engineer  who  typically  expends  much 
energy  incorporating  the  design  change  with  little  motivation  to  spend  any  additional  time 
and  energy  on  updating  manuals. 
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3PA  COSTING  TECHNIQUES 


Mr.  Theodore  3.  Post 
BioTechnology,  Inc. 
Falls  Church,  Virginia 


Presentation 


To  prepare  for  this  conference,  I  ran  a  literature  search  on  3PAs  and  cost.  To  my 
surprise,  the  search  generated  only  a  few  documents  (e.g.,  Booher,  1975;  Chenzoff,  1973; 
Rowan,  1973;  Defense  Analysis  Logistics  Office,  1977).  As  you  know,  more  documenta¬ 
tion  exists,  but  the  reports  tend  not  to  find  their  way  into  the  literature  (e.g., 
presentations  such  as  T.  Braid's  Life  Cycle  Cost  Model,  or  the  Air  Force's  Initial 
Technical  Order  Project  Findings* 2). 

After  this  less  than  fruitful  start,  I  chose  to  talk  about  the  topic  of  3PA  cost 
benefits,  hopefully,  in  a  way  that  is  somewhat  different  from  the  usual  treatment.  I 
might  note  that  my  intent  matches  a  point  Bob  Blanchard  made  in  his  opening  remarks 
this  morning;  namely,  that  EPICS,  the  sponsor  of  this  seminar,  is  seeking  tradeoffs 
relative  to  3PA  costs,  not  necessarily  the  nuts  and  bolts  of  costing. 

My  talk  will  include  the  three  major  topics  shown  in  the  first  slide.  Regarding  the 
first  topic  (What  do  we  have  now?),  I'll  be  brief  because  we'll  probably  be  talking  a  lot 
about  it  over  the  next  2  days.  On  the  second  topic  (What's  wrong  with  what  we  have?),  I'll 
point  out  the  costing  technique  shortcomings  which  I  believe  are  restraining  3PAs  from 
becoming  a  more  effective  system  development  tool.  The  third  topic  (What  do  we  need?) 
is  actually  included  as  part  of  the  second  topic. 


- -  ^ 

TOPICS  TO  BE  COVERED 


•  What  do  we  have  now? 

•  What's  wrong  with  what  we  have? 

•  What  do  we  need? 


Post:  Slide  1 


What  Do  We  Have  Now? 

The  second  slide  shows  two  of  the  main  characteristics  of  the  costing  technique  in 
current  use.  Two  examples  of  the  page-dependent  characteristic  are  shown  in  Slides  3  and 
4.  Slide  3  represents  the  book  plan  for  part  of  the  system  hardware  in  this  example,  the 
channel  subsystem.  The  left  column  of  the  slide  shows  subtitles  pertinent  to  the 


Personnel  communication. 


2Project  cancelled,  only  drafts  were  available. 
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hardware;  the  headers  show  the  different  types  of  pages  which  can  appear  in  a  book  or 
technical  manual  (TM).  The  cell  entries  represent  an  expert's  estimate  pages,  by  type,  for 
each  subtitle.  Sums  by  column  appear  at  the  foot  of  the  matrix. 3 


\ 

WHAT  DO  HE  HAVE  NOW? 

•  Page-dependent 

-  Bookplan  type 

-  Personnel  resource  type 

•  System-specific  input 

-  LSA  products,  l.e.,  TIM 

-  User-task  products,  l.e.,  NTIPS 

_ _ J 


Post:  Slide  2 


The  page  counts  for  the  book  plans  of  each  hardware  item  or  maintainable  unit  are 
summed  in  order  to  prepare  a  master  book  plan  illustrated  in  Slide  4.  Header  titles  do  not 
change,  but  the  left  column  now  contains  the  names  of  the  maintainable  units  rather  than 
the  names  of  the  sections  which  make  up  a  maintainable  unit's  documentation.  Again, 
sums  appear  at  the  foot  of  each  column  with  one  of  the  columns  showing  a  grand  total 
(i.e.,  359  pages).  At  this  point,  the  person  preparing  the  cost  estimates,  usually  a 
contractor's  employee,  relies  on  his  organization's  experience  and  records  to  develop  cost 
estimates.  These  estimates  are  usually  in  terms  of  costs  per  page  type  and  are  labor 
dependent  (e.g.,  the  hours  and  pay  rates  of  the  personnel  involved). 

Where  did  the  estimator  get  the  information  that  appeared  on  the  book  plans  (e.g., 
the  maintainable  units,  the  section  titles)?  Whether  the  item  being  costed  is  a 
conventional  TM  or  a  set  of  3PAs,  the  inputs  normally  come  from  the  logistic  support 
analysis  (LSA).  The  top  of  Slide  5  represents  the  topdown  breakdown  prepared  by  LSA 
(Slide  6  presents  part  of  a  more  realistic  presentation  of  a  breakdown).  The  topdown 
breakdown  serves  as  one  dimension  of  the  task  identification  matrix  (TIM)  that  LSA 
delivers  to  the  TM/3PA  organization  (see  the  mid-portion  of  Slide  5).  The  second 
dimension  of  the  TIM  consists  of  generic  functions  (or  descriptive  information)  that  could 
be  performed  by  system  operators  and  technicians  (see  the  lower  part  of  Slide  5).  The 
result  of  associating  a  generic  function  with  a  hardware  item  is  a  task.  The  center 
portion  of  Slide  5  represents  the  association  of  these  to  dimensions  to  form  system  tasks. 
LSA  determines  the  task  required  to  operate  and  maintain  a  system  and  documents  its 
finding  in  the  form  of  a  TIM  provided  to  the  TM/3PA  organization. 


3This  bookplan  was  prepared  for  a  TM  revision  effort.  The  parenthetical  numbers 
represent  the  page  count  for  the  original  TM  version. 
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CHANNEL  SUB-SYSTEMS 
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Post:  Slide  3 


Master  Book  Plan 
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Post:  Slide  4 


Post:  Slide  5 
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Post:  Slide  6 


The  user-TI  match  represents  another  system-specific  input  that  is  different  from  the 
book  plan  and  TIM  approaches.  Developed  as  part  of  the  Navy  technical  information 
presentation  system  (NTIPS),  the  match  uses  characteristics  of  the  user,  his  task,  and  the 
task  performance  environment  to  define  the  types  of  TI  required.  This  TI  definition 
includes  content,  format,  and  style,  as  well  as  delivery  medium  (e.g.,  paper,  CRT).  The 
bases  for  the  nonmedium  definitions  include  user  aptitude,  training,  experience,  and 
number  of  users.  The  source  data  for  these  bases,  for  the  most  part,  are  the  letter  sheets 
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(e.g.,  D  Sheets)  developed  by  the  LSA  portion  of  the  integrated  logistics  support  (ILS) 
program. 


Regardless  of  which  approach  is  used  to  plan  the  TM/JPA  effort,  the  bottom  line  is 
usually  page-dependent  and  based  on  cost  estimates  prepared  by  experts.  Slide  7  shows 
some  page-dependent  cost  data  developed  by  the  USAF.  The  left  column  of  the  table  lists 
the  various  types  of  JPAs  or  the  elements  of  a  TM.  The  column  heads  list  the  labor  types 
(and  hourly  rates)  involved  in  preparing  information  products.  The  body  of  the  table 
indicates  the  hours  and  costs  related  to  each  cell;  for  example  the  writer  cost  estimate 
for  one  page  of  job  guide  text  is  $47.50  (5  hours  at  $9.50  per  hour).  Totals  for  all  labor 
and  material  cost  estimates  appear  at  the  far  right  (e.g.,  the  text  portion  of  a  single  job 
guide  page  is  estimated  to  cost  $71.52). 
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What's  Wrong  with  What  We've  Got?  (Slide  8) 


The  current  costing  practices  have  been  criticized  on  several  counts;  for  example, 
the  process  is  unilaterial  in  that  the  expertise  required  to  prepare  cost  estimates  resides 
primarily  with  the  contractors.  "*  However,  the  criticism  I  have  does  not  relate  to  the  nuts 
and  bolts  of  cost  estimating.  (I  think  it's  pretty  good- -especially  the  approach  being 
developed  by  the  USAF.)  My  concern  relates  more  to  the  bases  for  determining  the  types 
of  JPAs  for  which  we're  preparing  cost  estimates.  Specifically,  our  present  cost 
estimates  are  based  on  inputs  we  receive  from  the  LSA  portion  of  the  ILS  program  (if  it  is 
performed),  and  from  inputs  we  receive  from  the  acquisition  manager  in  the  form  of 
specifications.  In  the  former  case,  tradeoffs  regarding  manpower,  personnel,  and  training 
are  supposed  to  have  been  performed  with  the  results  reported  on  the  LSA's  lettered  data 
sheets.  We  all  know  that,  even  if  they  are  performed  (and  frequently  they  are  not),  the 
tradeoffs  seldom  include  JPAs  as  an  active  element. 


WHAT'S  WRONG  WITH  WHAT  WE'VE  GOT? 


•  Subsystem  Optimization 

-  JPA  experts  are  not  represented  In  early 
trode  studies 

-  JPA  developer  keys  on  personnel  demands  of 
the  system  under  development  with  insufficient 
attention  to  the  Novy's  personnel  resources. 


Post:  Slide  8 


What  Do  We  Need? 

With  the  benefit  of  hindsight,  I  will  attempt  to  illustrate  the  types  of  involvement  I 
believe  JPA  developers  should  have  in  the  early  portions  of  systems  analysis.  These 
involvements  can  occur  as  a  result  of  adding  new  expertise  to  the  ILS/LSA  teams  or  they 
can  occur  as  independent  analyses  (e.g.,  the  Hardman  Project4 5  efforts  which  would  also 
need  expertise  modification  since  the  process  does  not  now  include  TMs  or  JPAs). 

Personnel  Trades.  The  process  and  methods  used  to  generate  manpower,  personnel, 
and  training  needs  are  biased  in  the  direction  of  demands  made  by  the  system  under 
development.  These  demands  are  documented  and  made  available  to  JPA  developers 
(among  others).  In  the  NTIPS  world,  these  demands  will  be  translated  into  TI  require¬ 
ments  via  methods  such  as  the  user-TI  match.  The  problem  I  see  is  that  insufficient 
attention  (sometimes  none)  is  paid  to  whether  the  manpower,  personnel,  and  training 


4Initial  Technical  Order  Project  Findings,  The  Air  Force  Logistics  Management 
Center,  1970,  p.  32,  33. 

sHardman  methodology  handbook,  Volume  1:  Executive  summary  (preliminary  draft). 
Washington,  DC:  Office  of  the  Chief  of  Naval  Operations,  November  1980. 
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resources  of  the  involved  military  are  sufficient  to  meet  these  demands.  All  too  often, 
new  systems  have  been  fielded  only  to  discover  that  the  levels  of  training  and  experience 
that  TI  developers  were  told  to  assume  are  far  less  than  expected.  TI  development 
efforts,  such  as  the  user-TI  match  and  validation  and  verification,  will  not  detect  and 
correct  these  faults  because  they  too  are  based  on  what  the  system  demands  rather  than 
what  the  military  can  provide.  Users  in  the  field  and  development  managers  criticize  the 
symptoms  rather  than  the  causes  (e.g.,  claims  that  the  TMs  or  JPAs  are  inadequate 
seldom  consider  the  source  data  available  to  TM  or  3PA  developers). 

My  recommendation  to  relieve  this  problem  is  to  have  JPA  developers  involve 
themselves  in  the  tradeoffs  that  should  occur  in  the  early  analyses  of  system  develop¬ 
ment.  There  are  those  who  claim  that  we  don't  have  the  knowledge  to  perform  these 
types  of  tradeoffs.  I  agree  that  we  don't  know  as  much  as  we'd  like  to  know,  but  we  are 
gaining  on  the  problem  with  projects  such  as  the  USAF  series  on  training  and  3PA 
tradeoffs  (e.g.,  HASTY-TASTY)  and  the  EPICS  program  that  is  sponsoring  this  seminar. 

Logistic  Trades.  ILS  personnel  perform  a  second  type  of  trade,  again  very  often 
without  the  benefit  of  TM  or  3PA  involvement.  This  trade  concerns  automated  test,  both 
on-line  and  off-line.  Early  automated  test  equipment  (ATE)  applications  (those  referred 
to  as  Turnkey  versions)  went  to  the  field  supported  by  technicians  or  operators  who  were 
expected  only  to  be  able  to  turn  the  ATE  on  and  off  and,  if  a  failure  was  detected,  to 
perform  simple  remove  and  replace  actions  in  order  to  return  the  system  to  an  up  status. 
The  technician's  job  was  scaled  down  in  terms  of  workload  (number  of  personnel)  and 
complexity  (skill  level,  training,  and  TM  needs).  The  military  promptly  met  these  scaled- 
down  requirements.  Unfortunately,  field  experience  with  Turnkey  ATEs  showed  that  the 
trades  failed  to  account  for  a  variety  of  technician  responsibilities  including  the 
following:  ATE  covers  only  a  part  of  the  hardware  and  man  must  cover  the  remainder, 
ATE  works  accurately  only  part  of  the  time  (sometimes  a  depressingly  small  part  of  the 
time),  and  man  must  have  the  resources  and  skills  to  back  up  the  ATE.  Because  the  trades 
did  not  anticipate  these  main  responsibilities,  technicians  were  not  available  in  sufficient 
numbers  nor  with  the  necessary  skills,  knowledges,  and  resources  (including  3PAs  as  well 
as  test  equipment)  to  perform  these  functions.  Relatively  large  orders  for  additional  TMs 
were  among  the  "get  well"  efforts  necessary  to  correct  these  Turnkey  problems. 

Again,  my  solution  is  to  involve  human  factors  and  3 PA  interests  in  the  early  trades 
which  relate  to  man's  responsibilities.  In  the  case  of  the  Turnkey  ATE,  this  involvement 
would  include  two  phases.  The  first  phase  involves  an  analysis  to  allocate  functions  to 
man  and  machine.  Such  an  analysis  performed  by  human  factors  representatives  will 
identify  the  detection-isolation  tasks  (e.g.,  from  the  TIM)  for  which  operators  and 
technicians  have  full  responsibility,  as  well  as  the  detection-isolation  tasks  for  which 
technicians  will  backup  the  ATE.  The  product  of  the  first  phase  is  developed  by 
translating  these  task  responsibilities  into  manpower,  training,  and  3PA  needs  (e.g.,  the 
demands  the  system  under  development  places  on  the  personnel  system).  The  second 
phase  of  the  recommended  involvement  is  the  discrepancy  analysis  referred  to  under  the 
personnel  trades  discussion  (e.g.,  analysts,  including  3PA  experts,  must  reconcile  any 
differences  between  the  personnel  demands  made  by  the  system  under  development  and 
the  personnel  resources  of  the  military).  Early  involvement  of  3PA  experts  in  such 
personnel  system  trades  (e.g.,  use  of  fully  proceduralized  3PAs)  could  well  have  captured 
the  benefits  of  reduced  skill  level  sought  unsuccessfully  by  the  planners  of  Turnkey  ATE. 
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Benefits  and  Recommendations 


I  believe  the  benefits  (Slide  9)  of  involving  JPA  expertise  early  in  system  analyses  are 
as  follows. 


r 
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A 

BENEFITS  OF  JPA  EXPERTISE  IN 
EARLY  SYSTEM  TRADES 

.  JPA  Visibility 

•  Corporate  Memory 

•  JPA  Cost  Perspective 


Post:  Slide  9 


Higher  Visibility.  At  present,  system  acquisition  managers  tend  to  view  JPAs  (and 
technical  manuals)  as  necessary  evils,  costly  products  that  come  along  after  early  trade 
studies  have  solved  all  the  real  problems.  Changing  this  attitude  will  require  demonstra¬ 
tions  to  show  that  JPA  expertise  can  be  a  contributing  element  of  early  trade  studies. 

Corporate  Memory.  The  doctrine  is  largely  in  place  for  early  trade  studies,  but  TMs 
t  and  JPAs  are  not  a  prominent  aspect  of  these  efforts.  There  is  no  repository  for  the 

results  of  relevent  studies,  no  organization  to  translate  evaluation  results  (such  as  EPICS) 
into  trade  practices,  and  no  organization  to  advocate  JPAs  as  an  element  of  early  trade 
studies. 

Perspective  of  3 PA  Costs.  The  cost  estimating  practices  being  applied  and  developed 
within  the  JPA  commuity  appear  adequate.  However,  in  my  opinion,  we  are  not  posing 
the  cost  question  in  the  proper  perspective  when  we  ask,  "How  much  will  it  cost  to 
produce  x  number  of  y  type  JPAs?"  The  larger  and  more  impressive  issue  should  be,  "Can 
a  system  acquisition  manager  use  JPAs  to  bridge  the  gap  betwen  his  system's  personnel 
demands  and  the  Navy's  personnel  resources?" 

Discussion 


Mr.  Post  suggested  that  the  JPA  community  follow  a  Project  Hardman  type  of 
approach.  It  was  suggested  that  proper  ILS  analysis  would  do  what  Mr.  Post  was  proposing 
and  that  Hardman  was  a  "fix-it"  approach  to  poor  ILS  work.  The  group  agreed,  however, 
that  ILS  was  usually  not  done  properly.  A  suggestion  was  made  that  Hardman  could  be 
utilized  as  a  technique  to  improve  ILS,  especially  in  the  area  of  tradeoff  analysis. 

Problems  with  ILS  were  discussed  in  terms  of  costs  for  LSA.  It  was  noted  that 
acquisition  managers  often  feel  that  much  of  the  data  generated  is  not  useful.  The  group 
agreed  that  we  are  "locked-into"  an  ILS  framework.  Further  discussion  centered  on  the 
problem  of  measuring  actual  performance  as  a  criterion  for  technical  information  quality. 
Inadequacies  with  secondary  criteria,  such  as  text  readability,  were  discussed  and  the 
group  agreed  that  recent  specifications  requiring  validation  with  a  target  population 
would  increase  the  quality  of  the  end  product. 
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COST  IMPACT  OF  3PA  DEVELOPMENT  PROCESS 


Mr.  Fred  L.  Hart 
Kinton, Inc. 
Alexandria,  Virginia 


Presentation 


In  my  approach  to  looking  at  3PA  costs,  I  decided  to  voice  them  under  the  context  of 
the  process  through  which  3PAs  are  developed.  I  chose  to  take  a  look  at  those  activities, 
and  what  the  cost  impacts  of  those  activities  are. 

I  think  the  acquisition  manager  doesn't  necessarily  have  to  specify  the  process  by 
which  the  3PAs  are  developed,  but  he  should  know  the  factors  that  affect  costs  of  this 
process.  One  of  the  things  that  occurred  to  me  when  we  were  talking  this  morning  is  that 
you  need  to  have  the  acquisition  manager  know  what  elements  are  involved  in  the 
manuals.  He  needs  to  know  how  these  elements  are  used.  We  talk  about  the  development 
of  these  things  and  we  talk  about  type  of  text  and  the  format  and  the  number  of  pages. 

We  might  want  to  look  at  it  in  a  more  general  context  from  the  standpoint:  Do  we  need 
theory  in  the  manual  and,  if  so,  why — what  would  be  the  use  for  it? 

Considering  the  tasks,  steps,  and  graphics  in  a  3 PA- -how  much  graphics  is  necessary? 

To  what  level  are  you  going  to  develop  your  3PAs?  To  what  level  do  you  want  to  define 
your  task?  This  has  a  very  significant  effect  on  the  cost  of  developing  these  products. 

Another  big  question  is  whether  or  not  the  system  exists.  It  is  much  easier  to  develop 
tasks  on  an  existing  system.  On  a  nonexisting  system,  at  what  point  in  the  development  4 

process  does  the  technical  information  procured  come  into  play? 

In  looking  at  the  steps  in  the  development  of  3PAs,  I  want  to  separate  3PAs  into  two 
categories:  straight  procedural  3PAs  and  what  I  call  functional  3PAs.  In  the  functional 
area,  I  include  troubleshooting  aids. 

In  the  procedural  3PA  area,  I  consider  straightforward  remove-and-replace  main¬ 
tenance  types  of  tasks.  The  first  step  involved  is  identifying  the  tasks.  Again,  if  you're 
going  to  develop  a  specification,  I'm  not  so  sure  you're  very  concerned  with  how  that 
identification  occurs.  However,  when  the  process  is  done,  you  still  need  to  know  that  90 
to  95  percent  of  the  tasks  have  been  identified.  At  this  point,  you  have  to  consider  your 
audience  and  which  tasks  you  are  going  to  train  and  which  you  are  going  to  devote  to  on- 
the-job  training.  For  the  "first-cut"  task  analysis,  my  experience  has  been  that  you  are 
going  to  use  existing  documentation.  The  system  doesn't  necessarily  need  to  exist  at  this 
point,  except  that  engineering  drawings  must  be  available. 

The  second  step  of  this  task  analysis,  which  I  think  is  extremely  important,  is  the 
tryout  of  the  procedures  on  the  equipment.  First  of  all,  the  availability  of  the  equipment 
is  a  problem  and,  second,  even  if  the  equipment  is  available,  "Are  they  going  to  let  you 
touch  it  or  take  it  apart  to  verify  your  procedures?"  It  has  always  been  my  experience 
that,  at  this  point,  there  is  not  enough  communication  between  the  documentation  and 
engineering  departments.  The  documentation  people  have  to  get  close  to  the  engineering 
people  to  convince  them  that  you  can  help  them  do  their  job,  especially  in  cases  where 
identification  of  a  procedure  finds  some  fault  or  problem  with  the  equipment. 

When  I  talk  about  validation,  I  mean  validation  on  the  actual  equipment  with  a  target 
audience.  I  think,  from  a  cost  effective  standpoint,  you  may  want  to  try  a  sampling 
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procedure  in  selecting  your  tasks.  The  bottom  line  at  this  point  is:  The  procedure  has  got 
to  work.  Again,  the  procuring  activity  should  look  at  the  most  economical  way  of  doing 
validation. 

The  procuring  agency  has  to  define  precisely  what  they  want  in  their  documentation. 
They  should  also  know  the  cost  impact  of  each  of  these  items. 

3PA  Cost  Elements  Identified  Through  3PA  Development  Process 

I.  Introduction. 

A.  Procurement  involves  two  types  of  3PAs. 

1.  Procedural. 

2.  Functional. 

B.  Costs  are  significantly  different. 

1.  When  systems  exist. 

2.  On  new  systems. 

II.  Procedural  3PA  Cost  Element. 

A.  Steps  in  process. 

1.  ID  tasks. 

a.  Tied  to  maintenance  philosophy. 

b.  Must  identify  all  tasks  (95-99%?). 

2.  First-cut  task  analysis. 

a.  Uses  existing  documentation. 

b.  Engineering  drawings,  etc. 

c.  Identifies  as  many  steps  as  possible. 

d.  Identify  graphics  to  extent  possible. 

3.  Tryout  on  equipment. 

a.  Identifies  all  steps. 

b.  Establish  all  graphics. 

4.  Draft  procedure  for  validation. 

a.  All  steps  with  graphics. 

b.  In  final  format. 

5.  Validation. 

a.  On  equipment. 

b.  Requirement  for  number  of  subjects. 
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6.  Final  preparation. 


a.  Incorporated  changes. 

b.  Prepare  camera-ready  copy, 

b.  Print. 

B.  Cost  effects  in  each  step  of  the  process. 

1.  Identify  tasks. 

a.  New  equipment. 

b.  Existing  equipment. 

c.  Formality  of  documentation  to  ensure  completeness. 

2.  First-cut  task  analysis. 

a.  Existing  equipment. 

(1)  Quality  of  documentation. 

(2)  Completeness. 

(3)  Availability  of  equipment. 

b.  New  equipment. 

(1)  Availability  of  documentation. 

(2)  Availability  of  SMEs. 

(3)  Availability  of  equipment. 

3.  Tryout  on  equipment. 

a.  Availability  of  equipment. 

b.  Environment  for  photographs. 

c.  Availability  and  use  of  target  population. 

4.  Draft  procedure  for  validation. 

a.  Virtually  no  external  constraints. 

b.  In-house  production  staff. 

c.  Good  time  for  IPR. 

5.  Validation. 

a.  Availability  of  equipment. 

b.  Availability  of  target  population. 

c.  Number  population  used  to  validate. 

d.  Sample  versus  total  validation. 

6.  Final  preparation. 

a.  Again  internal  to  contractor. 

b.  Function  of  production  system. 

c.  Camera-ready  copy  requirements. 

d.  Printing. 

e.  Printing  requirements. 
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III.  Functional  3PA  Cost  Elements. 


A.  Element  of  functional  3PAs. 

1.  Block  diagrams/associated  text. 

a.  Top-down  breakdown. 

b.  Complete  traceability  from  level  to  level. 

c.  Complete  interconnection  information. 

d.  Text  for  each  functional  block  at  each  level. 

2.  Schematics/associated  text. 

a.  Only  for  repairable  assemblies. 

b.  Redrawn  for  functional  layout. 

c.  Text  for  each  functional  grouping. 

3.  Troubleshooting  aids. 

a.  Big  area. 

(1)  Sympton/cause  tables  at  all  levels  of  system  on  down. 

(2)  Tree  charts  procedures. 

(3)  State  tables. 

(4)  Fully  proceduralized. 

(5)  Dependency  charts. 

(6)  Built-in  tests. 

(7)  Computer  diagnostics. 

b.  Aids  to  be  used  must  be  precisely  defined. 

B.  Steps  in  process. 

1.  Schematic  preparation. 

a.  Schematic  layout  by  writer. 

b.  Art  department  prepares. 

c.  Writer  prepares  text. 

2.  Block  diagram  preparation. 

a.  System  functional  analysis  for  top-down  breakdown  (used  for  both 
theory  and  troubleshooting). 

b.  Each  block  laid  out  by  writer. 

c.  Art  department  prepared. 

d.  Writer  prepares  text. 

3.  Troubleshooting  aids. 

a.  Varies  with  each  type. 

b.  Great  deal  of  analyses. 
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c.  Requires  art  support. 

d.  May  require  some  preparation  as  procedural  3PAs  along  with  analyses. 

C.  Cost  effects  of  functional  3 PA  processes. 

1.  Schematic  preparation. 

a.  Requires  engineering  drawings. 

b.  Access  to  SME. 

c.  Process  used  to  prepare  art. 

(1)  By  hand. 

(2)  Computep-aided. 

d.  Number  of  schematics  to  prepare. 

2.  Block  diagram  preparation. 

a.  Requires  complete  set  of  interconnection  data. 

b.  Time  to  perform  functional  analysis  obviously  increases  with  increase  in 

system  size. 

c.  Access  to  SMEs. 

d.  Access  to  equipment. 

e.  Process  used  to  prepare  art. 

3.  Troubleshooting  aids. 

a.  Access  to  test  equipment  to  be  used. 

b.  Access  to  equipment. 

c.  Access  to  SMEs. 

d.  Process  used  to  prepare  art. 

Discussion 


Mr.  Hart  described  task  analysis  as  an  initial  listing  of  tasks  with  as  many  steps  as 
possible  listed  in  their  proper  sequence.  Discussion  then  turned  to  the  difference  between 
task  analysis  and  procedure  writing  by  a  technical  writer.  The  consensus  was  that  there 
isn't  as  much  difference,  given  today's  practices,  as  there  used  to  be  in  the  past. 
Discussion  then  centered  on  the  topic,  "Is  task  analysis  worth  the  cost  and  to  what  level  of 
detail  do  you  do  it?"  Group  agreement  was  that  it  was  necessary  and  that  the  level  of 
detail  depends  upon  your  target  population. 

Discussion  of  the  number  and  type  of  subjects  needed  vs.  the  cost  involved  (for 
validation)  exacted  various  opinions.  Mr.  Hart  said  that  the  bottom  line  was,  "What 
assurance  of  quality  does  the  customer  want?"  Contractors  had  mixed  experiences  with 
validations;  the  favorable  experiences  came  from  those  who  said  that  the  validation  was 
planned  from  the  program  onset  and  contractor  personnel  conducted  the  validation. 
Don  Finegan  commented  that  the  Army  puts  a  target  audience  description  in  their  RFP. 

Discussion  of  the  readability  formulas  used  to  specify  reading  level  of  a  target 
audience  brought  out  points  about  their  nonvalidation  and  nonmeaningfulness  to  users. 
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The  group  agreed  that  some  measure  of  communicability  was  needed  and  that  readability 
formulas  were  not  the  best. 

The  advantages  and  disadvantages  of  block  diagrams  vs.  schematic  diagrams  were 
discussed.  The  general  consensus  was  that,  regardless  of  whether  block  diagrams, 
schematics,  or  MDCs  are  utilized,  these  items  are  high-cost  items  requiring  skilled 
writers  to  produce  them  and  skilled  technicians  to  utilize  them. 
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REDUCING  JPA  DEVELOPMENT  COSTS 


Dr.  Kay  Inaba 

Xyzyx  Information  Company 
Canoga  Park,  California 


Presentation 


Let  me  first  explain  my  perspective  on  this  problem.  I'm  approaching  this  from  the 
viewpoint  of  a  practitioner  as  opposed  to  a  researcher.  There  are  four  areas  that  we  have 
found  we  have  to  pay  quite  a  bit  of  attention  to,  with  some  areas  needing  more  attention 
than  others.  The  first  area  is  adjusting  the  format.  Over  the  years,  we  found  that  we 
really  haven't  had  to  adjust  the  format  all  that  much.  In  most  cases,  the  adjustments  are 
primarily  to  gain  the  acceptance  of  experienced  technicians.  We  find  that  we  never  have 
any  problem  with  inexperienced  technicians,  always  the  experienced  ones.  We  find  that 
these  changes  will  most  often  have  to  be  made  at  a  cosmetic  level  as  opposed  to  a  content 
level. 

The  second  area  is  overcoming  barriers  of  acceptance.  One  way  to  overcome  the 
"barrier"  of  the  experienced  technician  is  to  adjust  the  format.  Another  way  is  to  provide 
what  we  call  "system  explanations,"  or  theory  of  operation  of  the  equipment.  We  have 
found  that  many  experienced  technicians  tend  to  get  insulted  by  JPAs  and  feel  as  though 
you're  asking  them  to  do  a  "monkey  see,  monkey  do  type"  operation.  Therefore,  we've 
developed  a  parallel  set  of  documents  (i.e.,  to  "system  explanations").  These  documents 
tend  to  lend  credence  to  the  fact  that  management  is  interested  in  whether  the  technician 
knows  how  the  system  works.  Another  set  of  "barriers  to  acceptance"  is  the  system 
acquisition  manager.  Once  in  a  while,  we  run  into  an  enlightened  acquisition  manager; 
however,  they  are  the  exception  rather  than  the  rule. 

The  third  area  is  the  measurement  of  performance  for  JPAs.  As  long  as  people  aren't 
held  accountable  for  proper  performance  of  maintenance,  it  doesn't  matter  whether  JPAs 
are  used  or  not.  From  the  technician's  point  of  view,  having  JPAs  means  they  have  to  do 
something  different;  so  they  resist  this  change.  Things  change  somewhat  when  perfor¬ 
mance  is  measured.  Whether  or  not  the  technicians  are  held  accountable  for  their 
performance,  their  performance  becomes  visible  and,  therefore,  more  receptive  to 
accepting  help. 

The  fourth  area  is  cost.  In  my  experience,  cost  is  probably  the  biggest  single  barrier 
for  JPA  application.  Inevitably,  when  JPAs  are  considered,  cost  is  the  first  question- -es¬ 
pecially  if  a  technical  manual  already  exists.  My  experience  has  been  primarily  with 
industry.  I  had  a  recent  experience  with  a  major  defense  contractor  in  the  aerospace 
industry.  They  already  had  technical  documentation  and,  although  they  agreed  with  the 
value  JPAs  conceptually,  they  essentially  said  "forget  it"  when  we  got  to  the  topic  of 
cost. 

In  terms  of  cost,  I'd  like  to  separate  what  I  call  the  "front-end-analysis"  from  the  JPA 
generation.  I  will  address  both  topics,  but,  by  separating  them,  I  think  I  can  better  define 
how  we  have  been  able  to  stabilize  costs  in  each  area.  Given  that  distinction  between  the 
two  cost  phases,  we  feel  we've  been  able  to  stabilize  production  costs  primarily  through 
the  use  of  computer-aided  authoring  systems  that  we  call  computer-aided  technical 
information  preparation  system  (CATIPS).  Though  this  program  is  far  from  perfect,  it  has 
enabled  us  to  semiautomate  the  production  process.  This  program  is  really  an  evolution  of 
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the  manual  process  that  we  developed  for  presentation  of  information  for  maintenance 
organization  (PIMO),  and  we  eventually  fine  tuned  it. 

We  wanted  to  recognize  that  the  process  of  preparing  3PA  is  something  different 
from  the  process  ordinarily  used  to  prepare  technical  manuals.  We  recognized  that  you 
couldn't  give  a  bunch  of  technical  writers  a  new  specification  and  tell  them  to  "go  to  it" 
and  expect  them  to  meet  the  specifications.  There's  nothing  in  the  specifications  (that 
were  developed  10,  15  years  ago)  that  said  that  you  had  to  write  obscure  information.  If 
they  (the  writers)  could  have  done  it  properly,  they  would  have.  There  isn't  much  that  we 
do  in  3PAs  that  is  prevented  by  the  specifications.  It  is  practice  that  has  resulted  in  poor 
manual  preparation.  Given  that  it  is  practice,  getting  a  new  set  of  specifications  to 
writers  isn't  going  to  result  in  better  manuals.  So,  what  we  did  was  to  break  down  the 
process  of  writing--that  is,  converting  technical  engineering  type  data  into  3PAs--into 
steps  and  regroup  these  steps.  In  the  computer-aided  format  that  exists  now,  the  program 
took  us  about  2  years  to  develop. 

With  that  program,  I  break  down  cost  into  three  major  areas:  The  first  is  what  I  call 
"technical  management,"  which  is  such  things  as  interacting  with  subject  matter  experts, 
utilizing  technical  engineering  data,  and  basically  transforming  the  technical  information 
through  the  writing  process.  The  second  is  technical  personnel;  and  the  third,  production. 
Quite  often  I've  been  asked,  "What  would  happen  if  we  took  away  the  graphics  or 
illustration  portion  of  the  3PAs?"  My  opinion  is  that,  if  we  did  so,  only  a  20  percent 
reduction  in  cost  would  be  achieved.  In  other  words,  in  my  opinion,  there's  no  reason  not 
to  use  text  graphics,  if  you  can  reduce  the  cost  by  effective  management  of  the  technical 
data  rather  than  by  eliminating  the  graphics  per  se. 

Production  areas  that  we've  identified  for  major  improvement  are  graphics  editing 
and  interactive  generation  of  text.  When  I  say  "aid"  in  the  construction  of  text,  I  mean 
exactly  that.  I  don't  mean  text  processing  per  se.  Some  of  the  recent  studies  in  text 
processing  have  shown  that  about  60  percent  of  a  person's  time  is  spent  just  trying  to  get 
information  from  his  head  into  the  form  of  sentences  and  paragraphs.  In  using  text 
processors,  the  text  processor  usually  helps  in  "massaging"  the  text  after  it  has  been 
created.  In  CATIPS,  however,  we  put  most  of  our  effort  into  helping  the  writer  create 
the  text  in  the  first  place.  The  reason  is  very  practical.  The  high  cost  of  creating  text 
means  we  have  to  charge  the  customer  more  or  swallow  the  cost  ourselves,  thereby 
decreasing  the  probability  of  getting  the  contract. 

Given  such  a  method  of  handling  the  text,  we  found  that  the  major  cost  drivers  are: 
(1)  troubleshooting  and  (2)  quality  of  the  input  data.  An  area  that  I  feel  has  a  major 
impact  on  cost  is  the  decision  on  how  to  go  about  aiding  troubleshooting  or  handling 
troubleshooting  tasks.  If  you  decide  to  proceduralize  troubleshooting,  you  can  expect  to 
increase  the  number  of  pages  by  65  to  130  percent.  We  find  that,  if  you  do  it  properly, 
the  cost  per  page  isn't  that  much  greater,  although  it  is  greater  than  a  conventional 
manual.  The  big  factor  is  the  volume,  however.  Some  alternatives  may  be  to  provide 
logic  trees  or  similar  text-graphic  materials  to  highly  skilled  technicians.  Another 
alternative  is  simply  to  use  schematics.  The  use  of  particular  techniques,  of  course, 
depends  on  the  associated  training  program.  Some  espouse  intensive  training  and  using 
only  good  block  diagrams.  I  feel  that  this  area  could  have  a  significant  impact  on  the  cost 
of  3PAs.  Thus,  we  need  to  focus  on  how  to  handle  troubleshooting. 

The  second  area  is  the  quality  of  the  input  data,  and  this  leads  us  to  some  relevant 
myths.  First,  I  don't  believe  in  task  analysis  as  it  is  implemented  in  the  greatest 
percentage  of  cases.  It  seems  that  the  primary  orientation  is  to  crank  out  a  whole  lot  of 
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data  that  are  seldom  if  ever  used.  I  believe  that  if  you're  going  to  do  an  analysis  you 
should  "pinpoint"  the  efforts  to  produce  outputs  that  are  definitely  going  to  be  used. 

The  development  of  a  task  identification  matrix  is  essential,  but  if  you  look  at  the 
capability  that  is  required  to  develop  a  task  identification  matrix  from  scratch  (i.e.,  using 
only  engineering  data),  you'll  find  that  the  kind  of  analyses  required  is  not  normally 
considered  to  be  human  factors.  The  need  is  really  for  engineering  analyses.  Let's 
consider,  for  example,  one  that  determines  whether  a  particular  piece  of  equipment  needs 
to  be  inspected  or  checked  at  a  given  frequency.  Now,  that's  the  type  of  analyses 
required  to  develop  a  task  identification  matrix  and  that  I  would  consider  to  be  an 
engineering  analysis. 

Another  myth  that  I've  heard  is,  "JPAs  have  to  be  perfect."  To  some  extent,  that's 
true.  However,  if  you  try  to  make  each  JPA  perfect,  the  cost  is  going  to  go  "sky  high." 
We  find  it  is  important  to  orient  properly  the  users  (i.e.,  to  deal  realistically  with  minor 
deficiencies  in  the  JPA).  Essentially  any  JPA  could  be  made  more  perfect  but,  in  most 
cases,  the  JPAs  are  useful  and  usable  as  is.  Attempting  to  achieve  a  "perfect"  JPA  is 
going  to  drive  the  cost  up  tremendously. 

Other  problems  related  to  task  analysis  include  conducting  essentially  the  same 
analysis  over  and  over  again.  There  are  considerable  similarities  between  systems.  For 
example,  most  electronic  systems  have  quite  similar  skill  demands.  I'm  not  saying  that 
one  shouldn't  conduct  a  thorough  task  analysis,  but  that  one  should  only  analyze  areas  for 
which  the  skills  are  new,  as  opposed  to  repeating  the  same  analysis  and  hiding  the  useful 
information  with  a  multitude  of  pages. 

In  taking  a  look  at  some  of  the  analyses,  I  can't  overemphasize  the  importance  of 
engineering  analyses,  as  currently  known  in  the  form  of  LSAs.  I  firmly  believe  that  we 
should  provide  human  factors  support  for  the  people  who  are  involved  in  the  development 
of  the  LSARs.  Just  because  the  quality  of  the  data  (LSARs)  is  not  good  at  this  time,  we 
should  not  continue  to  reject  these  data  as  the  human  factors  community  has  done  for  so 
many  years.  By  rejecting  the  LSA  efforts,  we've  taken  full  responsibility  for  generating 
the  data.  In  fact,  the  human  factors  community  does  not  have  the  capability  to  generate 
this  type  of  data  from  scratch.  We  usually  get  the  data  by  querying  the  subject  matter 
experts.  We  even  have  the  audacity  of  calling  the  effort  a  form  of  analysis,  when  in  fact 
it  is  not.  Sometimes  we're  lucky  and  get  good  subject  matter  experts  and  get  good 
results.  However,  more  often  than  not,  we  are  not  lucky.  I  contend  that  it  is  much  more 
realistic  to  make  the  LSA  people  accountable  for  the  data  and  help  them  improve  their 
process. 

We  have  learned  to  separate  the  price  for  "front-end"  analysis  and  JPA  generation. 
We  tell  the  customer  that  the  front-end  data  that  they  provide  will  directly  determine  the 
quality  of  the  final  product.  If  it  is  poor,  then  the  final  product  will  be  very  usable,  but 
still  poor.  We  let  them  know  that,  if  they  want  help  in  cleaning  up  these  data,  then  we 
will  provide  such  help.  We  let  them  know  that  it  will  cost  them  exactly  the  same  for  the 
production  whether  or  not  they  choose  to  clean  up  the  input  data,  but  the  responsibility 
for  quality  rests  with  them. 

We  do  not  always  succeed  in  getting  the  customer  to  accept  the  responsibility  for  the 
technical  content.  Life  becomes  much  more  difficult  in  such  cases,  but  we  no  longer  try 
to  correct  the  problems  ourselves.  At  the  minimum,  we  share  the  responsibility- -es¬ 
pecially  since  we  try  to  make  our  position  clear  from  the  outset. 
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We  also  give  the  customer  the  option  of  changing  the  format  of  the  3PA  because  the 
material  is  stored  electronically  and  the  changes  can  be  made  easily.  It  does  cost  a 
certain  amount  of  extra  money  to  make  these  changes;  however,  the  electronic  storage  of 
the  data  reduces  these  costs.  Keep  in  mind  that  it  is  a  buyer-driven  market  and  we  feel 
that  we  must  meet  the  needs  of  the  customer. 

Regarding  troubleshooting  materials,  my  experience  has  been  that  90  percent  of  the 
people  who  conduct  the  failure  mode  effect  analyses  (FMEAs)  and  dependency  analyses 
don't  know  how  to  do  them  correctly.  This  directly  affects  the  quality  of  input  data  for 
troubeshooting  tasks.  One  possibility  for  improving  these  data  is  to  let  the  producer  know 
that  we  are  going  to  cross-check  the  analysis  (not  a  full-blown  analysis,  but  a  limited 
check).  A  better  approach  is  to  agree  upon  a  process  and  teach  this  process  to  the  LSA 
specialist.  However,  this  is  an  expense  that  many  organizations  are  not  willing  to  incur. 

Discussion 


Initial  discussion  of  barriers  to  3PA  implementation  highlighted  a  potential  role  for 
the  government  in  helping  contractors  "get  a  handle"  on  their  cost  estimates,  while 
reducing  the  "fear"  of  3PA  production  in  relation  to  other  technical  information 
presentation  methods. 

Dr.  Inaba  briefly  discussed  the  use  of  block  diagrams  and  certain  techniques  that  may 
be  used  for  "cosmetic  purposes."  Dr.  Inaba  was  asked  how  cost  affects  competition  for 
project  awards.  He  explained  the  difference  in  costing  between  "strictly  3PA  develop¬ 
ment"  projects  and  those  where  the  technical  information  3PA  was  purchased  noncompet- 
itively.  He  was  asked  whether  he  would  purchase  a  3PA  package  that  was  10  times 
costlier  and  20  percent  more  effective  than  another  package.  Dr.  Inaba  explained  that 
the  procurement/development  system  was  not  set  up  to  reward  the  person  delivering  the 
more  cost/effective  system;  therefore,  cost  vs.  effectiveness  would  not  even  be  con¬ 
sidered  in  the  procurement  stage.  This  is  why  they  (Xyzyx)  break  down  cost  into  two 
stages--(l)  front-end  development,  and  (2)  production- -and  instill  the  quality  responsi¬ 
bility  on  the  customer. 

Discussion  then  turned  to  identifying  the  point  of  responsibility  for  conducting  up¬ 
front  data  analysis  of  troubleshooting  tasks.  The  need  for  FMEA  or  dependency  analysis 
was  expressed  by  Dr.  Inaba  with  the  concern  that  the  customer  does  not  usually  bear  the 
responsibility  for  these  analyses.  Discussion  then  focused  on  the  responsibility  for  these 
analyses  within  an  organization.  The  design  engineer,  as  opposed  to  the  technical  writer, 
was  targeted  for  doing  these  analyses,  although  the  actual  practice  of  this  doctrine 
differed  and  the  burden  of  these  analyses  on  the  engineer ng  people  was  considered  to  be 
substantial.  The  discussion  of  background  data  collection  and  organization  identified  the 
technical  publications  people  as  the  heaviest  users  of  LSA  data  and  noted  that,  while  the 
merit  of  the  LSA  concept  could  be  established,  there  were  serious  problems  with  the 
implementation  of  the  process. 
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SUMMARY  OF  COST  FACTORS  FOR  ONE  JPA  DEVELOPMENT  TASK 


Mr.  William  Conroy 
Raytheon  Service  Company 
Ventura,  California 


Presentation 


What  I  have  done  for  my  presentation  is  to  put  together  a  summary  of  a  single  task 
that  we  performed  up  in  Ventura.  I  will  show  how  we  went  about  estimating  cost  and 
what  happened  as  the  task  evolved.  The  task  was  a  simple  one  involving  the  development 
and  production  of  maintenance  requirement  cards  (MRCs).  The  tasks  were  defined  and 
the  maintenance  requirement  description  was  already  written  for  22  tasks  that  were 
involved. 

One  of  the  things  I  had  to  take  into  consideration  was  that  my  personnel  who  were 
available  to  work  on  this  job  were  not  familiar  with  the  equipment.  By  "not  familiar,"  I 
mean  they  didn't  understand  the  nuts  and  bolts  of  it,  although  they  very  quickly  learned 
what  it  was  supposed  to  do  and  what  type  of  equipment  it  was.  Another  factor  was  that 
the  equipment  was  available  locally  and  it  was  on  the  station  at  the  Naval  Ship  Weapon 
Systems  Engineering  Station  (NSWSES),  10  miles  from  our  facility.  The  equipment  was 
available  to  us  any  time  we  wanted  to  go  and  take  a  look  at  it  and  that  also  meant  that  we 
could  validate  it  locally.  Specifically,  this  equipment  was  a  new  plasma  display 
microprocessor  terminal  on  the  NATO  Seasparrow  Surface  Missile  System. 

The  tasks  that  were  defined  were  all  remove-and-replace  procedures.  The  22 
procedures  were  defined  in  another  technical  manual  written  by  another  Raytheon 
organization.  Considering  our  discussion  today,  I  have  to  admit  that  I  didn't  even  ask  if 
there  was  an  LSA  available.  What  I  would  like  to  do  (after  this  meeting)  is  find  the  LSA 
data  and  compare  the  two. 

Since  I  knew  the  equipment  and  the  people  that  were  involved,  this  is  the  kind  of 
weighting  that  I  put  on  the  types  of  people  who  would  be  producing  the  MRCs  (see  Slide 
1).  I  can't  say  that  this  is  a  normal  rating  because  every  job  will  be  different.  In  this 
particular  case,  I  knew  the  capabilities  of  the  writers  who  were  going  to  be  involved  in  the 
job  and  was  able  to  use  that  information.  Then,  of  course,  I  assumed  that  we  would  edit, 
type,  and  illustrate. 

To  get  this  number  of  pages,  it's  obvious  that  we  assumed  two  pages  of  text  to  one 
page  of  illustration.  In  rating  it,  I  was  really  hedging  because  I  knew  I  would  make  a 
savings  as  some  of  the  art  work  would  be  redundant.  On  the  other  hand,  not  knowing  the 
guts  of  the  equipment,  there's  always  a  possibility  that  illustrations  that  we  did  not  think 
of  would  be  required.  Because  of  the  time  frame  of  this  task,  the  engineering  drawings 
were  not  available  and  we  were  working  from  an  existing  preliminary  technical  manual, 
certainly  not  written  to  any  specification  at  this  point.  The  illustrating  factor  is  large 
because  we  had  to  illustrate  and  draw  them  by  looking  at  the  actual  equipment,  and  this  is 
more  time  consuming.  In  this  case,  we  probably  got  a  better  product  because  the 
engineering  drawings  did  not  exist  and  we  looked  at  the  actual  equipment. 
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ESTIMATING 


•  MAINTENANCE  REQUIREMENT  DESCRIPTION  -  PROVIDED 

•  EXISTING  PERSONNEL  NOT  FAMILIAR  WITH  EQUIPMENT 

•  EQUIPMENT  AVAILABLE  LOCALLY 

•  VALIDATION  TO  BE  PERFORMED  LOCALLY 
ESTIMATE 


•  WRITING 

44  TEXT  PAGES 

X 

8.6  HRS 

•  EDITING 

44  " 

99 

X 

.3  HRS 

•  TYPIST 

44  " 

99 

X 

.7  HRS 

•  ILLUSTRATING 

22  " 

99 

X 

8.6  HRS 

374 

13 

31 

187 

606 


Conroy:  Slide  1 

Slide  2  shows  the  general  sequence  of  the  work  that  we  do  and  the  order  in  which  we 
usually  do  it.  We  do  some  research  and  try  to  locate  any  engineering  data,  if  we  can. 
Then,  we  would  identify  what  art  we  would  generate,  and  the  tools  required  for  the  job. 
In  other  words,  these  are  things  that  should  have  been  on  an  LSA.  At  this  point,  the 
writer  puts  together  rough  text  that  is  typed  and  then  proofread  while  we  develop  the 
artwork.  Writers  and  illustrators  were  able  to  go  to  the  equipment  and  work  on  the 
procedures  by  looking  at  the  equipment.  At  this  point,  we  develop  a  worksheet;  however, 
it  is  not  the  formal  MRC  worksheet.  We  view  it  for  general  accuracy;  for  example,  if  we 
take  something  apart,  we  check  if  it  is  put  back  together  again  and  general  "editing-type" 
things.  We  try  to  consider  the  sailor  as  much  as  possible.  Most  of  the  people  that  work 
for  me  in  Ventura  are  ex-sailors  and  they  understand  the  working  environment  and  the 
sailor.  They  understand  the  language  that  has  to  be  provided  here.  For  this  review,  we 
put  the  information  in  the  regular  MRC  format  using  word  processing  throughout.  After 
this  initial  production,  we  did  a  validation  utilizing  a  writer,  "customer-provided"  person, 
and  a  sailor.  They  allowed  us  to  remove  quite  a  few  of  the  modules,  and  we  actually  did 
such  and  put  them  back  in. 
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•  IllUSTMTI 


Conroy:  Slide  2 


[Note.  At  this  point,  a  discussion  occurred  concerning  the  particular  background  for 
this  study  and  Slides  3  and  4  were  presented.  Results  of  the  study  showed  that  inaccurate 
task  definition  data  changed  the  complexity  of  the  MRCs,  requiring  decreased  preparation 
time  with  increased  time  for  correction  of  task  data.  The  complexity  and  amount  of 
required  research  was  highlighted  in  Slide  5  as  the  primary  cost  driver.  Level  of  detail 
and  the  requirements  for  artwork  were  presented  in  Slide  5.  Mr.  Conroy  commented  that 
safety  was  a  particular  consideration,  given  that  these  people  are  working  in  an  unfamiliar 
environment.  The  presentation  text  continues  with  the  "Safety"  item  on  Slide  5.  ] 

The  working  environment  must  be  considered.  Quite  often  in  the  past,  when  we  were 
far  removed  from  the  actual  equipment,  we  missed  many  of  these  considerations.  For 
example,  equipment  location  is  a  critical  environmental  factor  when  one  piece  of 
equipment  is  four  decks  and  two  doors  away  from  another  piece  of  equipment,  and  the  guy 
is  constantly  running  back  and  forth  to  look  at  a  particular  signal.  In  other  words,  the 
writers  who  are  working  on  the  particular  JPA  should  have  this  knowledge  of  the  overall 
environment  and  what  it  takes  to  get  the  job  done.  Consideration  of  the  supply  system  is 
necessary  when  writing  troubleshooting  tasks.  Often,  you  write,  "if  this  card  is  bad 
replace  it,  and  if  this  other  card  is  bad,  replace  that  one."  Well  the  first  card  might 
involve  a  certain  amount  of  delivery  time,  and  the  second  card  might  take  even  longer.  I 
think  we  have  to  remember  this  in  developing  troubleshooting  aids.  In  other  words,  the 
cards  aren't  right  there  for  them  to  just  take,  insert,  and  replace.  In  some  systems,  a 
ready-spares  cabinet  is  available  with  parts,  but  in  many  of  the  newer  systems  I  have  not 
seen  this  to  be  true. 
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ACTUAL 


\ 


TASK  WAS  REDEFINED  |  OF  THE  22  ORIGINAL 
PROCEDURES  IT  WAS  DECIDED  THAT  6  WERE  NOT 
REQUIRED  BUT  4  OTHER8  WERE  -  AND  THE8E  OTHER8 
WERE  MORE  COMPLEX  THAN  ORIGINALS  - 


ACTUAL 


WRITING 

33 

TEXT  PAGES 

X  8.6  HRS 

a 

281 

EDITING 

33 

n 

X  .3  HRS 

E3 

10 

TYPIST 

33 

## 

if 

X  .7  HRS 

- 

23 

ILLUSTRATING 

(UNIQUE) 

14 

H 

i# 

X  8.6  HRS 

119 

433 

ACTUAL  HOURS  USED  ■«  609  HRS 

Conroy:  Slide  3 
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COST  DEPENDS  ON  COMPLEXITY  AND  AMOUNT  OF  RESEARCH  REQUIRED 

ESTIMATES  FOR  AN  AVERAGE  PAGE  OF  TEXT  ARE: 


RESEARCH:  l 

8  HOURS 

WRITE  DRAFT:  f 

TYPE  WORKSHEET: 

0.15  HDURS 

PROOF  RE  AD/ED  IT: 

0.1  HOURS 

TYPE  PRELIM  MRC  PAGE: 

0.15  HOURS 

PROOFREAD: 

0.1  HOURS 

TYPE  CORRECTIONS: 

o.i  Houns 

VALIDATE: 

0.6  HDURS 

TYPE  CORRECTIONS: 

0.1  HOURS 

PROOF: 

0.1  HOURS 

TYPE  CORRECTIONS: 

0.1  HOURS 

Q.C.: 

0.1  HOURS 

AVERAGE  PAGE:  SR.  WRITER 

8.5  HOURS 

prooF/edTt 

0.3  HOURS 

TYPING 

0.7  HOURS 

AYERAGE  FULL  PAGE  ILLUSTRATION: 

{TRACE  ENQ.  DRAWING/ 

REDUCE) 

8  HOURS 

REPRODUCE/PASTE  UP 

0.5  HOURS 

_ y 


Conroy:  Slide  4 
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CONSIDERATIONS 


LEVEL  OF  DETAIL 

•  REQUIRES  MORE  CONSIDERATION  OF  ENVIRONMENT 

•  SHIP  CLASS  DIFFERENCES 

ARTWORK 

•  SHOULD  CLEARLY  IDENTIFY  PHY8ICAL  CONTEXT 

•  MUST  SHOW  VIEWS  THAT  ARE  POSSIBLE 

SAFETY 

•  MUST  BE  AN  EVEN  8TRONGER  CONSIDERATION  THAN 
WITH  NON-EPICS  8AILOR8 

WORKING  ENVIRONMENT 

•  DISTANCES  BETWEEN  WORK  8TATIONS 

•  SUPPLY  8YSTEM 


Discussion 


Conroy:  Slide  5 


Mr.  Conroy  defined  the  22  MRCs  as  having  more  text  than  illustrations,  but  were 
expected  to  be  on-the-job  aids  (i.e.,  JPAs). 

Peculiarities  of  the  development  background  behind  Mr.  Conroy's  example  were 
noted;  namely,  that  technical  drawings  and  technical  manuals  were  not  purchased  with  the 
original  system.  Although  one  conference  participant  could  not  understand  why,  another 
commented  on  the  cost  savings  by  not  having  to  update,  catalogue,  purchase  such 
information  initially.  Group  discussion  of  the  advantages/disadvantages  of  "automatic" 
technical  manual  purchasing  continued,  with  the  predominant  feeling  that  decisions  should 
be  made  on  a  case-by-case  basis. 

Discussion  then  centered  on  Mr.  Conroy's  "working  environment"  considerations  in 
Slide  5.  The  availability  of  spare  cards  for  use  in  troubleshooting  tasks  was  discussed,  as 
well  as  the  utilization  of  spare  parts  for  troubleshooting.  Mr.  Conroy  summarized  that 
there  must  be  more  thought  involved  in  developing  troubleshooting  procedures,  given  that 
spare  parts  are  not  instantly  available.  He  emphasized  the  point  that  the  user's 
environment  must  be  carefully  considered  by  writers. 
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COMPUTER-BASED  TECHNICAL  ORDER 
ASSESSMENT  METHOD  (PAGES) 


( 


Ms.  Rosemarie  d.  Preidis 
U.S.  Air  Force  Human  Resources  Laboratory 
Wright-Patterson  Air  Force  Base,  Ohio 


Presentation 


Currently  we  have  a  computer-based  technical  order  assessment  method  called 
PAGES.  It  is  operated  in  an  interactive  mode.  These  algorithms  are  used  to  estimate  the 
content  of  troubleshooting  and  nontroubleshooting  data  for  both  flightline  and  shop 
maintenance.  PAGES  is  meant  for  application  during  the  conceptual  and  validation  phase 
to  provide  relative  content  and  quantity  estimates.  PAGES  may  be  used  to  estimate 
either  conventional  or  task-oriented  manuals.  PAGES  estimates  pages  for  both  electrical 
and  mechanical/hydraulic  systems. 

These  algorithms  are  dependent  on  a  knowledge  of  the  system,  the  number  of 
subsystems,  and  the  quantity  of  line  replaceable  units  and  shop  replaceable  units  within 
the  target  system.  The  user  can  directly  input  this  data  at  the  terminal  or  he  may  use  the 
equipment  configuration  data  bank  of  the  reliability  and  maintainability  model. 6 

The  PAGES  model  provides  page  estimates  for  12  different  page  types.  These  page 
types  are: 


1. 

Narrative. 

2. 

Half-tone  art. 

3. 

Half-tone  explosion. 

4. 

Electronic  line  art. 

5. 

Exploded  line  art. 

6. 

Fault  isolation  chart. 

7. 

Fault  isolation  schematic 

block. 

8. 

Access  line  art. 

9. 

Fault  isolation  schematic 

flow. 

10. 

Fault  isolation  schematic 

mechanical /hydraulic, 

11. 

dob  guide  narrative. 

12. 

dob  guide  illustration. 

PAGES  algorithms  can  best  be  described  by  example.  The  following  example  involves 
the  algorithm  to  predict  the  content  of  a  fault  isolation  manual  to  support  the  task- 
oriented  approach  to  flightline  troubleshooting.  The  total  pages  are  calculated  as  follows: 

No.  pages  =  2  fault  isolation  charts/subsystem 
+  2  fault  isolation  charts/LRU 
+  1/2  page  narrative/LRU 
+  2  access  line  art  pages/LRU 

+  2  fault  isolation  schematic  block  pages/subsystem 
+  1  fault  isolation  schematic  flow  page/LRU. 


6The  reliability  and  maintainability  (R&M)  model  is  an  average  value  model  that 
provides  outputs  of  R&M  parameters  in  a  form  useful  for  initial  studies  and  tradeoff 
analyses  in  early  acquisition. 
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The  cost  estimation  portion  is  presently  a  manual  operation.  It  is  based  on  a  page 
type/cost  area  matrix  (Slide  1).  Cost  data  are  presented  in  labor  hours  and  unloaded 
costs.  This  matrix  is  used  to  present  the  time  and  cost  estimates  for  the  various  type 
pages  considered. 

The  technical  order  content  and  cost  estimation  methodology  is  based  on  a  F-16 
technical  order  study  that  took  place  during  May-September  1978.  At  the  request  of  the 
F-16  system  program  office  (SPO),  the  Air  Force  Human  Resources  Laboratory  (AFHRL) 
performed  a  content  and  cost  analysis  of  the  technical  order  requirements  for  the  F-16 
production  aircraft.  The  purpose  was  to  provide  "baseline"  information  to  F-16  SPO 
procurement  personnel  for  use  in  contract  negotiations  concerning  the  purchase  of 
production  aircraft  technical  orders.  AFHRL  developed  algorithms  for  predicting  content 
of  technical  orders  for  task  oriented  organizational  data  and  the  conventional  interme¬ 
diate  and  depot  data.  The  algorithms  were  based  on  the  page  count  and  page  type  of 
F-15,  F-16  (FSD)  technical  orders.  The  approach  used  in  collecting  cost  data  was  to 
obtain  cost  estimates  from  several  contractors  for  developing  the  different  types  of 
pages.  The  basic  guidelines  to  the  contractors  were  that  engineering  drawings  were 
available  and  that  a  front-end  task  analysis  was  complete.  The  composition  with  type 
page  costs.  The  information  provided  by  the  algorithms  and  cost  data  was  instrumental  in 
significantly  reducing  the  negotiated  purchase  price  of  the  F-16  production  aircraft 
technical  orders. 

Addendum 


Ms.  Preidis  picked  a  hypothetical  electrical  system  and  inserted  these  data  into  her 
PAGE  algorithm.  Her  hypothetical  system  contained  32  subsystems  and  77  LRUs,  which 
yielded  a  total  of  552  pages.  She  then  explained  the  generation  of  the  cost  estimation 
matrix,  which  was  done  by  contacting  three  contractors  and  asking  them  how  many  labor 
hours  would  be  involved  given  a  certain  number  of  page  and  illustration  requirements. 

Ms.  Preidis  explained  that  these  algorithms  are  part  of  a  larger  overall  project  called 
Project  1959.  The  project  encompasses  training  requirements,  technical  data,  reliability, 
and  maintainability  in  terms  of  looking  at  one  design  vs.  another  design.  This  particular 
page-estimation  package  is  concerned  with  comparisons  of  the  technical  data  require¬ 
ments  of  various  designs.  She  also  explained  that  these  algorithms  have  not  been 
validated  yet  and  their  purpose  at  present  is  to  develop  a  base-line  estimate  of  the 
number  of  pages  and  amount  of  cost  involved. 

Discussion 


Ms.  Preidis  indicated  that  electrical  systems,  as  specified  in  paragraph  1  of  her 
paper,  include  electronic  systems.  Subsystems  were  defined  as  being  based  upon  a  work 
breakdown  type  of  coding  system. 

As  a  result  of  the  technical-order  study  conducted  in  1978,  a  reduction  in  cost  for  the 
F-16  proposal  was  realized.  Discussion  revealed  that  a  major  source  of  that  reduction  was 
in  reduced  hours  estimated  for  certain  work  areas.  Discussion  of  the  models'  weaknesses 
included  the  fact  that  a  writer  may  prepare  a  block  diagram  as  an  initial  aid  to  guide  his 
writing;  however,  this  time  is  not  considered  as  illustration  time  even  though  he  has  drawn 
the  diagram.  The  conclusion  was  that,  for  the  model's  purposes,  this  was  a  minor 
drawback  and  the  work  breakdown  categories  could  never  be  "black  and  white." 
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COST-DIRECT  LABOR  DATA  -  SOURCE  I 
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COST-DIRECT  LABOR  DATA  -  SOURCE  I  (continued) 


Discussion  of  tailoring  the  algorithms  to  specific  user  types  (i.e.,  the  same  type  of  job 
guide  might  be  different  for  different  groups)  brought  out  points  that  a  job  guide  (as  used 
in  the  algorithm)  already  indicates  a  user  type,  and  very  specific  user  differences  would 
only  have  a  small  influence  on  page  numbers.  A  discussion  of  troubleshooting  procedures 
brought  out  an  earlier  point  that  troubleshooting  vs.  nontroubleshooting  procedures 
significantly  changed  page  volume,  but  once  a  decision  was  made  selecting  trouble¬ 
shooting  as  a  method  of  choice,  the  variation  was  small. 

A  general  comment  that  these  algorithms  could  be  better  applied  to  a  higher  level 
system,  such  as  an  F-16  aircraft,  than  to  a  smaller  radar  system  was  mutually  agreeable. 
A  question  concerning  the  number  of  equipment  design  changes  and  the  effect  of  the 
number  of  changes  on  the  3PA  text  revealed  that  data  of  this  type  were  available  in 
AFHRL  data  files  and  that  these  changes  definitely  influenced  cost.  The  number  of 
changes  are  an  uncertainty  factor  usually  included  in  costing  and  also  included  in  the  hour 
estimates  for  the  AFHRL  estimates.  The  number  of  changes  are  usually  estimated  by 
using  historical  data  to  compare  past  and  present  systems.  A  point  was  made  that  the 
customer  must  know  if  this  uncertainty  factor  for  equipment  changes  was  included  in  a 
cost  estimate  because  it  could  account  for  a  wide  discrepancy  between  two  contractor 
bids  on  the  same  procurement. 
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3PA  PRICE  ESTIMATING:  IMPACT  OF  3PA'S  NATURE  AND 
CIRCUMSTANCES  EXPECTED  FOR  ITS  DEVELOPMENT 


Mr.  Fran  Rahl,  3r. 
Westinghouse  Electric  Corporation 
Hunt  Valley,  Maryland 


Presentation 


3PA  price  estimating  as  currently  accomplished  is  a  combination  of  art  and  science 
that  I  approach  from  several  directions,  depending  on  circumstances.  On  an  existing 
system  already  documented  in  a  traditional  format,  a  fairly  accurate  task  count  can  be 
made.  Estimating  norms  based  on  historical  performance  or  time-and-motion  study  are 
then  applied  to  determine  page  count  and  cost  by  labor  type.  On  developmental  systems 
or  other  undocumented  systems,  task  analysts  first  estimate  the  number  of  tasks  and  then 
employ  page  norms  to  determine  the  cost.  Where  insufficient  time  or  equipment 
definition  exists,  a  rough  order-of-magnitude  estimate  can  be  made  by  assuming  the  3PA 
price  will  be  a  certain  proportion  of  the  total  program  price  (where  such  knowledge 
exists).  Finally,  a  "management  calibration"  number  will  always  be  developed  using  rules 
of  thumb,  comparisons  with  similar  previous  jobs,  or  general  feeling  for  the  expected  level 
of  effort. 

Regardless  of  the  method(s)  employed,  innumerable  variables  of  two  general  cate¬ 
gories  (Slide  1)  affect  the  estimator's  viewpont  and  ultimately  the  3PA  price  estimate: 
The  nature  of  the  3PA  and  the  expected  circumstances  under  which  it  will  be  developed. 

(  \ 

JPA  PRICE  IS  A  FUNCTION  OF: 

•  THE  NATURE  OF  THE  JPA  ITSELF 

•  THE  CIRCUMSTANCES  SURROUNDING  ITS 
DEVELOPMENT 

/ 

V _ ) 

Rahl:  Slide  1. 

The  nature  of  the  3PA  (Slide  2)  can  readily  be  determined  by  looking  at  it  for  physical 
appearance  characteristics  such  as  text/graphics  mix,  size,  color,  gross  volume,  density, 
etc.  Of  course,  the  detailed  specification  requirements  are  a  function  of  user  profile, 
task  types  and  quantities,  equipment  type,  and  specification  items  reflecting  current 
thinking  on  usability.  Slides  3  through  10  are  examples  of  some  of  the  forms  Westinghouse 
has  employed  in  the  past  few  years.  These  samples,  arranged  from  lowest  intended  user- 
skill  level  to  highest,  illustrate  the  effects  of  desired  appearance  on  cost,  when  compared 
on  Slide  11.  Unskilled  samples  A-D  (Slides  3-6)  have  the  highest  proportion  of  illustration 
labor  as  compared  to  analytic  and  writing  labor.  In  some  instances,  this  labor  mix  results 
in  a  lower  price  per  page;  however,  the  need  for  more  pages  may  offset  the  savings. 
Semi-skilled  sample  E  (Slide  7)  has  a  higher  mix  of  writing  to  illustrating  than  unskilled; 
however,  fewer  pages  are  required.  Skilled  sample  F  (Slide  8)  has  a  high  writing  content 
and  has  the  greatest  information  density  of  the  technical  data  forms  normally  thought  of 
as  3PAs.  Skilled  samples  G-H  (Slides  9,  10)  are  conventional,  have  the  highest 
concentration  of  writing  labor,  and  have  the  fewest  pages  per  task. 
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THE  NATURE  OF  THE  JPA  IS  DEFINED 
IN  ITS  PHYSICAL  APPEARANCE, 
AND  IS  A  FUNCTION  OF: 


•  USER  PROFILE 

•  TASK  TYPE,  QUANTITIES 

•  EQUIPMENT  TYPE 

•  SPEC  REQUIREMENTS 


TEXT/GRAPHICS  MIX, 

.  DATA  DENSITY, 
GROSS  VOLUME,  ETC. 


Rahl:  Slide  2 
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EXAMPLE  A  •  UNSKILLED;  DIRECT  VISUAL  ACTION  CUES 
WITH  MINIMUM  TEXT  •  (STANDARD  SIZE) 

J 


Rahl:  Slide  3 
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4  of  16 
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EXAMPLE  B  -  UNSKILLED  DIRECT  VISUAL  ACTION  CUES  WITH 
MINIMUM  TEXT  -  (POCKET-SIZE) 


Rahl:  Slide  4 
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-SAMfLI- 


REMOVING  PRESSURE  VESSEL 


nt  <XM*n  DO  Cap'lVI t  Kfim  1 1 1  ext  covert 

12.  3.  4.  6)  Wtih  no.  3  crow  l«p  tcrewdriver 
R#mo v*  covert 


DmennKi  com  uU«  (61  •<  both  wx» 
with  11/16  inch  end  7/16  inch  open  end 
wfOnthot. 


BD.Konncct  MMf •  hornm  (7)  end  •Mcincel 
P*«f  (61 


Oitconneci  five  luLm  (9.  10.  11,12,  13) 
with  3/16  Inch  open  and  wrench. 


EXAMPLE  C  • 


UNSKILLED;  FULLY  PROCEDURALIZED 
WITH  CUED  TASK  STEPS 


TEXT 

y 


Rahl:  Slide  5 


46 


47 


IM  ?-2JW)-?SS.?0  1-3-2 


Volutin  lit 

IV*«.  7-11.  Tnti.  7- Ml 


EXAMPLE  E  -  SEMI-SKILLED;  FULLY  PROCEDURALIZED 
WITH  SUPPORTIVE  CUES  (FRAME  PAGE) 


Rahl:  Slide  7 
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TO  1C-14U-2-4JQ-4 

REMOVE  AMO  IHSTALL  FUEL  PUMP 

Remove  And  Stow  Low 
Presiur*  Switch  Asaembly. 

13  Disconnect  clemp  13)  with 
ettached  load  from  tube  (21 

14  Disconnect  coupling  nuts 
(1)  at  both  ends  of  tube  (2) 
Remove  end  cap  tube 

15  Remove  two  nuts  (41 
securing  switch  hrecket  (6j  to 
bottom  of  fuel  pump  (7) 

16.  Remove  entire  switch 
assemhly  (5)  with  attached 
electrical  lead 

17  Stow  switch  assembly  (5) 
clear  of  fuel  pump  If 
necessary  tie  or  tape  switch 
to  engine  structure 

is  Disconnect  clamp  i 0 |  with 
•tteched  tube  19)  by  removing 
locknut  1 10)  from  bottom  of- 
pump 


'N 


EXAMPLE  F 


SKILLED;  FULLY  PROCEDURALIZED  TEXT  KEYED 
TO  A  SINGLE  CUE/LOCATOR  (FACING  PAGE) 


Rahl:  Slide  8 
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1.  Monuo!  y  trip  riot  tor  If  riwctor  w*N 
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AMTIOPArfolKAMSUm 
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c  MonuoNy  ttatl  vwtaor. 
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Example  H.  Conventional-Tabular 


Rahl:  Slide  10 
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JPA  TYPE  SUMMARY  AND  COST  FACTORS 


%  OF  EFFORT 

PAGE  RATIO 

USER  PROFILE 

RECOMMENDED  FORMAT 

TASK  ANAL. 

WRITING 

GRAPHICS 

CONVENT  VS  JPA 

UNSKILLED 
(APPRENTICE  LEVEL) 

STM  -  7TH  RGL 

DIRECT  CUE/RESPONSE  HIGHLY  ILLUSTRATED 
(PRESENTED  AS  ACTION’  CUES)  - 
MINIMUM  TEXT  (RESPONSES'  WRITTEN 

IN  SHORT  CONCISE  MANNER  AND 

KEYED  WITH  EACH  TASK  STEP. 

(SAMPLE  A  O) 

30% 

10% 

60% 

1:10 

SEMI  SKILLED 

(TECH  SCHOOL  ORAD  AND/OR 
OJT) 

7TM  •  BTH  ROL 

SEPARATE  TASK  STEPS  IN  PROCEDURALIZED 
ORDER  •  ORAPMIC  (CUES)  DEPICTING 

LOCATION  OF  PARTS  RATHER  THAN 
•ACTIONS’  AS  USED  FOR  UNSKILLED 
PERSONNEL  (SAMPLE  E) 

10% 

30% 

40% 

18 

SKILLED 

(SPECIALIST -LEVEL) 

ATM  12TM  RGL 

FULLY  PROCEDURAL1ZED  TFXT  KEYED 

TO  A  SINOLE  ORAPHIC  CUE 

EXPLODEO  OR  LOCATOR  VIEWS  OF 

EOUIPMENT  COMMONLY  USED  IN  THIS 

TYPE  OF  FORMAT.  (SAMPLE  F) 

30% 

50% 

20% 

14 

SKILLED 

^SPECIALIST-LEVEL) 

BTH  •  14TH  ROL 

CONVENTIONAL  .  EXTENSIVE  NARRATIVE 
NON-PROCEDURAUZED  -  ILLUSTRATIONS 

USED  FOR  COMPLEX  PROCEDURES  ONLY 
GENERALLY,  FORMATTED  AS 

EXPLODED  VIEWS  (NONCUED). 

(SAMPLE  O  N) 

00% 

10% 

1:1 

_ J 


Rahl:  Slide  11 
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Conventional  technical  manuals  serve  as  the  standard  for  comparison  in  the  page 
ratios  (data  density)  shown  in  the  chart.  These  kinds  of  data  are  also  the  least  expensive 
for  several  reasons:  (1)  Negligible  task  analysis  is  required,  (2)  procedures  can  often  be 
lifted  from  engineering  source  data  with  little  tailored  professional  writing,  and  (3) 
illustrations  are  usually  developed  from  equipment  design  data  and  do  not  require  action 
cues. 

This  discussion  implies  that  some  differences  in  price  can  be  expected  between  the 
various  3PA  formats  available,  on  a  per-page  basis.  Most  likely,  however,  variances 
between  the  approaches  on  a  total  job  basis  will  be  insignificant.  The  difference  between 
the  price  of  conventional  manuals  and  any  JPA  of  similar  scope  are  quite  significant.  The 
bottom  line — 3PA  price — is  relatively  insensitive  to  format,  once  the  decision  has  been 
made  to  use  a  proceduralized  approach. 

What,  then,  are  the  big  cost  drivers? 

Earlier  in  my  presentation,  I  indicated  there  are  two  general  categories  of  variables 
affecting  price.  The  nature  or  appearance  is  one.  The  other,  and  by  far  most  significant 
category,  includes  all  of  the  circumstances  surrounding  the  3PA  development  (Slide  12). 


CIRCUMSTANCES  SURROUNDING 
DEVELOPMENT  OFTEN  AFFECT  PRICE 
MORE  THAN  MEASURABLE  FACTORS 
(FORMAT,  MEDIA,  DEPTH,  ETC.): 

•  COMPETITION 

•  UNCERTAINTY 

•  EXISTENCE  AND/OR  AVAILABILITY  OF  HARDWARE 

•  EXISTENCE  AND/OR  AVAILABILITY  OF  ENGINEERING 
DATA 

•  DESIGN  STABILITY 

•  PROGRAM  TYPE 

•  PREPARER’S  ORGANIZATIONAL  PECULIARITIES 

•  CUSTOMER  TYPE  (PERSONALITY,  KNOWLEDGE,  ETC.) 

•  CUSTOMER  TYPE  (GOV’T.  AGENCY,  COMMERCIAL,  ETC.) 

•  TIMING/SCHEDULE 

_ _ _ J 


Rahl:  Slide  12 

The  existence  and  strength  of  competition  affects  3PA  pricing  in  several  ways.  First, 
the  cost  to  the  producer  (less  general  and  administrative  costs,  cost  of  money,  and  profit) 
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is  more  closely  estimated.  Fewer  contingency  funds  are  included;  less  project  follow-on  is 
anticipated.  The  low  ends  of  the  quoting  norm  scales  are  used.  Frills  are  specifically 
excluded  with  proposal  caveates.  Then,  within  the  bounds  permitted  by  the  accounting 
standards,  practices,  and  business  objectives,  the  profit  figure  is  set  in  a  further  attempt 
to  develop  a  competitive  price. 

Uncertainty  is  a  factor  in  many  programs,  particularly  developmental  programs. 
Estimators  are  often  asked  to  price  JPAs  for  hardware  that  has  not  yet  been  designed. 
There  is  a  natural  tendency  to  assume  the  worst  case  under  these  circumstances,  and  this 
attitude  is  reflected  in  the  estimates.  The  same  problem  holds  true  for  other  support 
elements  such  as  test  equipment,  the  design  of  which  is  speculative  of  the  prime 
equipment  design. 

Of  course,  this  uncertainty  does  not  exist  on  programs  where  design  has  progressed 
well  into  full-scale  engineering  development  or  production  phases.  The  estimators  have 
the  opportunity  to  examine  the  equipment  or,  at  least,  the  drawings.  Having  the  hardware 
available  during  the  3PA  development  is  extremely  beneficial  and  allows  for  the  best-cost 
conduct  of  the  job.  This  is  particularly  important  where  the  3PA  employs  a  high  level  of 
visual  cueing.  Possession  of  the  equipment  allows  for  actual  performance  of  the  task, 
during  which  photographs  of  each  cue  are  taken.  Photo-line  conversion  then  provides  the 
most  accurate  and  least  expensive  illustrations  possible. 

Availability  of  engineering  data  at  the  right  time  is  very  important  to  cost-effective 
3PA  development.  It  is  an  absolute  requirement  on  the  many  programs  where  actual 
hardware  availability  is  limited  or  nonexistent.  In  addition,  data-dependent  analysis,  such 
as  head-book  tradeoff,  level  of  repair,  and  others,  are  impossible  without  a  fair  amount  of 
engineering  data. 

For  the  most  part,  the  availability  of  the  engineering  data  is  a  function  of  program 
timing  and  design  stability.  Prior  knowledge  or  anticipation  of  this  kind  of  problem 
affects  the  3PA  estimator  in  an  obvious  way. 

The  program  type  can  have  a  dramatic  effect  on  price.  Program  type  refers,  for 
example,  to  the  Air  Force  buying  equipment  from  a  prime  contractor  where  a  technical 
manual  is  included.  In  this  case,  the  technical  manual  could  be  purchased  at  a  lower  cost 
from  the  subcontractor,  because  the  prime  adds  in  a  certain  amount  for  managing  and 
procuring  the  technical  data.  After  this  markup,  the  prime  will  sell  the  technical  data  to 
the  government  or  procuring  agency.  The  technical  data  end  up  being  high  cost  items  at 
that  point,  largely  due  to  extra  markup.  It  may  be  worth  it  to  the  services,  however,  to 
have  the  program  managed  centrally. 

Preparer's  organizational  peculiarities  also  affect  price.  Labor  costing  methods, 
driven  by  the  accounting  system,  result  in  all  or  part  of  the  task  analysis  being  charged  at 
an  engineering  hourly  rate  or  the  opposite  situation  could  exist  wherein  all  of  the  analysis 
and  writing  are  costed  at  a  lower  technical  writing  hourly  rate.  Under  the  worst 
circumstances,  all  of  the  labor  types  contributing  to  the  3PA  development  and  production 
could  be  priced  at  one  rate,  well  above  the  expectation  for  production  service  labor. 
Certainly,  the  method  of  LSA  development  (who,  where,  when,  how)  will  affect  cost, 
particularly  where  engineering  department  personnel  do  the  D-sheets,  which  must  then  be 
redone  by  engineering  writers. 

An  organization  whose  quality  standards  are  built  on  years  of  military  experience 
may  not  be  able  to  produce  JPAs  at  any  other  (less  costly)  level,  as  might  be  required  to 
compete  in  a  commercial  or  foreign  market. 
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Slide  12  has  two  bullets  for  customer  type,  one  for  the  responsible  individual  person 
and  the  other  for  the  customer  organization  type.  In  one  of  the  previous  presentations, 
there  was  a  great  deal  of  discussion  about  the  procuring  specialist  and  the  need  for 
training  that  individual.  I  am  amazed  at  how  quickly  a  naive  procurement  specialist 
becomes  an  expert  (in  his  or  her  mind)  in  JPA  technology.  In  a  typical  scenario,  dealing 
with  this  individual  on  an  initial  contract  leaves  a  lasting  impression  that  will  cause 
follow-on  work  to  be  priced  accordingly.  This  is  less  of  a  problem  where  rigid 
specifications  and  definitive  statements  of  work  exist.  The  other  customer  type  bullet 
refers  to  customer  organizational  types  such  as  Air  Force,  Army,  Navy,  commercial,  etc. 
Costs  are  always  higher  on  Army  JPAs  due  to  multiple  reviews  by  numerous  people  and 
use  of  color;  commercial  or  foreign  JPAs  may  be  far  less  expensive  due  to  less  stringent 
requirements. 

Anticipated  schedule  problems,  either  protracted  or  condensed,  will  affect  price. 
The  estimator  will  accommodate  the  former  by  inefficient  man-loading  and  the  latter  by 
overtime  premiums.  He  or  she  will  also  consider  time  phasing  of  the  JPA  with  hardware 
and  engineering  data  development. 

The  conclusions  (Slide  13)  to  be  drawn  from  this  discussion  are  that  the  most 
important  cost  drivers  on  a  JPA  program  are  the  most  difficult  to  model.  Estimates 
under  these  circumstances  are  based  in  experience  and  anticipation.  While  cost  models 
have  and  continue  to  be  developed,  their  proper  application  is  in  comparative  analysis  for 
tradeoffs.  In  this  light,  they  could  serve  the  useful  purpose  of  the  LCC/LSC  models  whose 
outputs  do  not  truly  represent  life  cycle  cost  or  logistic  support  cost,  but  do  show  the 
relative  effects  of  design  change  options,  maintenance  concepts,  etc. 

Discussion 


Initial  discussion  on  the  detail/number  of  graphics  associated  with  a  user  skill  level 
brought  out  the  point  that  the  same  type  of  graphic  could  be  used  for  an  unskilled  person 
as  well  as  for  a  skilled  person.  Discussion  continued  on  information  type  and  the  format 
appropriate  for  different  skill  levels.  It  was  agreed  that,  quite  often,  changes  were  made 
as  a  result  of  customer  preference,  given  that  clear  guidelines  for  text/graphics  data 
format  (based  upon  skill  level)  do  not  exist.  The  use  of  computer  graphics  was  described 
as  an  excellent  cost-reducer  for  "2-D  type"  graphics  (i.e.,  flowcharts,  troubletrees,  etc). 

During  discussion  of  the  usefulness  of  cost  models,  mention  was  made  that  the  models 
would  represent  responsibility  to  someone  (i.e.,  "you  want  this  change- -here's  how  much  it 
will  cost").  Further  discussion  of  technical  data  costs  within  the  life  cycle  cost  of  a 
system  led  to  some  controversy  regarding  the  initial-preparation  cost  relevant  to  total 
cost  of  technical  data.  Discussion  of  JPA  costs  for  digital  equipment  included  reference 
to  the  fact  that  repetition  of  equipment  models  could  be  utilized  to  reduce  cost;  however, 
equipment  complexity  tended  to  increase  cost.  Group  consensus  was  that  technical 
manual  cost  was  a  significant  part  of  total  life  cycle  costs. 
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CONCLUSIONS: 


©  JPA  PRICES  ARE  GREATLY  INFLUENCED 
BY  FACTORS  OTHER  THAN  THOSE 
DEMONSTRATED  IN  THE  APPEARANCE  OF 
THE  FINAL  PRODUCT. 

©  THE  EFFECTS  OF  MANY  OF  THESE 
FACTORS  ARE  DIFFICULT  TO  PREDICT 
QUANTITATIVELY. 

•  PRICE  MODELS  WOULD  LIKELY  BE  OF 
MOST  USE  IN  PERFORMING  TRADE-OFFS 
(A'  LA  LCC)  AS  OPPOSED  TO  ESTIMATING 
FINAL  COST. 


Rahl:  Slide  13 
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IMPACT  ON  JPA  PROGRAMS  OF  SCHEDULES, 
STAFF  RESOURCES,  AND  CONTRACT  REQUIREMENTS 


Mr.  John  Weber 
Lockheed-California  Company 
Burbank,  California 

[Note.  In  the  initial  part  of  Mr.  Weber's  presentation- -as  outlined  in  Slide  l--he 
detailed,  from  a  historical  perspective,  his  personal  experience  with  JPA  programs  at 
Lockheed-California.  ] 


LOCKHEED-CALIFORNIA  COMPANY 
JOB  PERFORMANCE  AID  (JPA)  PROGRAMS 


•  CP-140  (CANADA) 

•  92  JPA* 

•  1276  PAGES  OF  TEXT  AND  GRAPHICS 

•  COMPLETED  1981 

•  P-3  (NAVAIR  JPA  PILOT  PROGRAM) 

•  10  JPA*  (PROP  &  ENGINE) 

•  219  PAGES  OF  TEXT  AND  GRAPHICS 

•  COMPLETED  1982 

•  P  3  (NAVAIR  LOGMOD/.'PA  PILOT  PROGRAM) 

•  3  JPA*  (AIR  CYCLE  SYSTEM) 

•  59  PAGES  OF  TEXT  AND  GRAPHICS 

J 

Weber:  Slide  1 


•  IN-WORK 

_ 


When  I  talk  about  schedule  (Slide  2),  I'm  really  talking  to  the  people  from  the 
government  here  and  not  to  the  industry  people,  because  they  know  what  a  lot  of  these 
problems  are.  Schedules  can  control  many  things  and,  most  importantly,  they  can  control 
who  will  respond  to  an  RFQ  and  even  who  is  going  to  be  the  winner.  I  can  get  an  RFQ  in 
the  mail  and  put  together  all  the  inputs  from  various  departments  and  hand  it  to  a 
contracts  administrator  within  2  weeks  if  I  really  push.  Then,  I  can  sit  back  for  3  to  6 
months  and  wait  for  a  proposal  to  go  through  the  finance  and  contracts  mill.  The  problem 
is  that  the  relative  scope  of  the  money  amount  for  the  technical  order  data  is 
insignificant  to  the  contracts  people  in  relation  to  the  hundred-million-dollar  type 
contracts  with  which  they  deal.  The  fact  that  I  need  this  funding  to  continue  what  I'm 
doing  doesn't  make  that  much  difference  to  them  in  perspective.  So  you  learn  how  to 
plead  with  these  people  to  get  your  contract  pushed  through. 
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SCHEDULE 


'\ 


•  CUSTOMER  CAN  CONTROL  WHICH  CONTRACTORS  WILL  RESPOND  TO  RFQ 

•  TIME  AVAILABLE  TO  CONTRACTOR  FOR 
•  RFQ  REACTION  TIME 

•  PRODUCTION 

•  FINANCE 

•  CONTRACTS 

•  PRODUCTION  DEPARTMENT  RESOURCES 

•  AVAILABLE  PERSONNEL 

•  VENDOR  AVAILABILITY 


•  FORMAT  REQUIREMENTS 


Weber:  Slide  2 


I  have  to  think  very  carefully  about  the  resources  that  I  have.  The  vendors  are 
difficult  to  deal  with  because  they  typically  give  you  a  long  time  estimate  for  getting 
done  what  you  request.  I  have  to  look  at  who  is  available  when  costing  this  out,  and  I  have 
to  look  at  the  average  experience  level  of  my  people.  If  I  have  to  bring  someone  in, 
there’ll  be  a  problem  of  even  finding  them  because  the  word  is  out  in  the  technical  writing 
community  that  in  JPA  production  you  have  to  follow  a  very  rigid  format  in  your  writing. 
The  key  is  to  attract  the  new  JPA  writer  by  challenging  him  and,  then,  we  have  to  train 
him.  This  training  is  going  to  take  an  average  of  500  hours;  therefore,  it  had  better  pay 
off.  During  the  training  period,  we  don!t  expect  a  lot  from  the  writer  in  preparing 
material  whether  it  be  intermediate  products  or  the  JPA  itself.  Finally,  when  the  new 
JPA  writer  goes  to  work,  you  hope  for  25  to  35  percent  decrease  in  time  per  page  over  the 
first  6  months.  Thus,  finding  people,  keeping  them  working  at  a  set  pace,  and  finding 
people  that  can  do  this  sort  of  work  day  after  day  is  very  important. 

There’s  another  factor  affecting  our  department  resources.  People  have  found  out 
that  we  take  a  very  systematic  approach  and  expect  them  to  produce  products  that  will 
work.  In  going  through  this  behavioral  task  analysis,  you  can  see  everything  as  far  as  what 
the  writer  did,  to  whom  they  talk,  and  how  they  came  up  with  their  writing.  When  a 
writer  finally  goes  through  the  validation  process,  he  must  experience  getting  feedback 
from  a  sailor- -this  feedback  may  be  negative.  Another  point  is  that  when  we  have  a 
slight  downturn  in  JPA  business,  many  people  jump  out  of  the  group.  Typically,  these 
people  are  very  successful  in  other  departments. 

In  regard  to  Slide  3  and  the  continuity  of  JPA  programs,  this  is  where  it’s  really  at  for 
me.  All  I’m  saying  is  that,  if  we  (the  JPA  staff)  go  out  of  business,  then,  it’s  like  "killing 
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the  last  of  a  species."  It  is  very  important  for  me  as  a  developer  to  have  people  available 
who  are  trained  and  who  have  the  proper  mental  attitude  required  for  this  kind  of  work.  I 
don't  think  the  personnel  mix  is  terribly  important  if  you're  talking  about  a  new 
specification  or  a  new  JPA  program  because  everybody  has  to  learn  that  from  the  start. 
For  the  new  developer,  however,  I  believe  very  strongly  in  on-the-job  training.  Once  you 
get  the  writer  to  the  point  where  he  can  write  successfully,  you  let  him  go.  Start  him 
working  with  a  task  he  understands,  like  changing  a  spark  plug  on  a  car.  That  training 
seems  to  work  well,  and  pretty  soon  he  is  able  to  carry  the  writing  load  on  his  own. 


•staff 


•  CONTINUITY  OF  JPA  PROGRAMS 

•  STAFF  SIZE  AND  MIX 

•  RATIO  OF  EXPERIENCED  TO  NEW  JPA  DEVELOPERS 

•  JPA  DEVELOPER  CANDIDATE  AVAILABILITY 

•  IN-HOUSE 

•  OUTSIDE 

•  TRAINING  REQUIREMENTS  FOR  NEW  PROGRAM 

•  EXPERIENCED  DEVELOPER 

•  NEW  FORMAT 

•  NEW  SPECIFICATIONS) 

•  NEW  USER  DESCRIPTION 

•  NEW  DEVELOPER 

•  CLASSROOM 

•  OJT 

•  INSTRUCTOR  SUPERVISOR 


•  SCHEDULE  SENSITIVE 


Weber:  Slide  3 


Slide  k  shows  the  contract  requirements.  I  believe  all  these  things  are  sensitive  to 
the  schedule.  The  schedule  will  decide  what  happens  internally.  The  level  of  detail  will 
impact  cost  to  a  degree,  but  will  not  really  cause  a  whole  lot  of  change.  You  have  to 
know  more  details  about  the  system  than  the  level  of  detail  to  which  you're  writing.  By 
source  data,  I  mean  that  you're  starting  out  with  blueprints,  existing  TOs,  and  all  of  the 
second  level  or  overhaul  manuals.  I  really  "press  home"  to  my  people  that  there's  no  point 
in  not  going  over  and  talking  to  a  design  engineer  and  obtaining  more  source  data.  The 
important  thing  is  to  put  a  trail  on  the  source  of  your  information — where  it  came  from.  I 
include  in  the  contract,  as  part  of  the  Navy's  responsibilities,  provisions  for  providing 
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equipment  and  personnel  for  verification  and  validation.  Today,  we  are  very  labor- 
intensive  to  the  point  where  we  are  introducing  very  many  errors  in  the  data  when 
corrected.  I  think  that  because  this  production  process  is  such  a  manual,  "people- 
involved"  process,  we  need  some  of  these  machines  to  get  us  away  from  some  of  these 
errors. 


•  CONTRACT  REQUIREMENTS 


t 


•  LEVEL  OF  DETAIL 

.  INTERMEDIATE  PRODUCTS 
•  JPA* 

•  AVAILABILITY  OF  SOURCE  DATA 

•  COOPERATION  OF  CUSTOMER 

•  USE  OF  EQUIPMENT  BY  CONTRACTOR 

•  USE  OF  CUSTOMER  PERSONNEL  BY  CONTRACTOR 

•  LABOR  INTENSIVE  VS  COMPUTER  AIDED 

•  TEXT  DEVELOPMENT 

•  GRAPHICS  DEVELOPMENT 

•  PRINT  READY  NEGATIVES 


•  SCHEDULE  SENSITIVE 

Weber:  Slide  4 


Discussion 


Mr.  Weber  explained  that  the  92  3PAs  prepared  in  the  CP- 140  program  were  for 
maintenance  "remove-and-repiace"  type  of  tasks.  Mr.  Weber  was  asked  to  elaborate  on 
the  effects  of  noncontinuity  in  3PA  program  funding.  He  explained  that  the  3PA  team 
would  dissolve  and  members  would  be  scattered  throughout  the  company.  The  start-up 
time  and  learning  curve  for  a  new  team  would  be  costly  in  terms  of  time  and  money. 

Discussion  then  focused  on  the  graphics  process  and  different  interactions  between 
the  technical  and  graphics  personnel  at  various  companies.  The  use  of  a  graphics 
coordinator  was  mentioned  as  a  mediator  between  the  graphics  and  technical  personnel. 
Quality  of  the  final  product  was  discussed  as  a  significant  cost/time  element.  "Short-cut" 
production  methods  of  producing  art  and  graphics  would  quite  often  be  acceptable  to  the 
customer,  but  not  to  contractor  in-house  quality  assurance  (Q/A)  people.  The  extra  effort 
in  raising  the  quality  of  the  product  could  be  substantial  in  terms  of  time  and  money,  but 
have  minimal  effect  on  the  perceived  quality  and  utility  of  the  final  product. 
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Mr.  Weber  elaborated  on  his  process  of  training  JPA  writers  by  giving  them  an  initial 
writing  task  on  a  well-known  procedure  and  determining  their  ability  to  think  and  express 
their  ideas  in  a  logical  format.  He  stated  that  individuals  who  cannot  initially  do  this 
usually  do  not  improve  to  the  point  of  being  cost  effective.  The  type  of  person  who  excels 
at  JPA  writing  may  be  an  engineer,  ex-flight-line  mechanic,  or  instructor,  with  no 
particular  background  predispositioning  successful  writers. 
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ROLE  OF  JPA  ACQUISITION  MANAGER 


Mr.  Reid  P.  Joyce 
Applied  Science  Associates 
Valencia,  Pennsylvania 


Presentation 


It  appears  that  the  developers  of  JPAs  have  made  some  reasonable  progress  towards 
refining  the  technology  and  increasing  the  probability  of  accurately  estimating  cost  for 
what  seems  to  me  to  be  the  "simpler  stuff,"  the  kind  of  practitioner  tasks  that  Kay  Inaba 
talked  about.  I  have  decided  to  go  back  over  the  notes  that  I  have  taken  here  in  the  past 
day  or  so  and  highlight  things  and  thoughts  that  were  part  of  my  original  discussion. 

I'm  little  bit  distressed,  although  not  really  surprised,  at  how  much  lack  of  detail  has 
been  given  to  one  of  the  more  serious  problems — that  is,  the  system  acquisition  manager. 
We  have  spoken  pretty  lightheartedly  here  about  "hobnailed  boots,"  but  have  always 
danced  rather  gingerly  away  from  the  problem  as  if  it's  not  solvable.  I'm  not  convinced 
that  it's  not  solvable,  although  I  don't  have  any  definitive  answers  at  the  present  time. 

I'd  like  to  go  through  some  of  the  presentations  given  here  and  highlight  some  points. 
Don  Finegan  started  by  presenting  a  study  that  the  Army  has  currently  underway.  He 
tried  to  see  where  existing  specifications  need  to  be  revised,  and  where  they  have  at  least 
permitted  cost  and  volume  excesses  within  the  limits  of  the  existing  specifications. 
During  this  meeting,  several  people  have  mentioned  similar  experiences  with  specifica¬ 
tions  failing  to  prevent  excesses  like  that.  It's  clear  that,  although  the  specifications 
provide  much  guidance,  they  still  permit  an  amount  of  stupidity.  A  JPA  acquisition 
manager  who  is  not  particularly  experienced  either  in  JPAs  or  acquisitions  can  make  many 
errors  that  are  permitted  by  the  specifications. 

Sam  Rainey  talked  about  the  need  to  get  people  thinking  in  terms  of  system 
ownership  and  to  educate  system  acquisition  managers  in  ways  of  procuring  systems  that 
people  will  use.  It's  pretty  clear  that  the  reward  structure  in  that  whole  procurement 
process  is  not  aimed  at  long-term  good.  It  really  does  ignore  the  notion  of  life  cycle  cost 
as  affected  by  the  kinds  of  things  that  we're  concerned  with.  We  really  don't  have  any 
control  over  the  establishment  of  JPA  requirements  that  are  going  to  solve  the  problem  of 
long-term  system  ownership.  As  Ted  Post  says  regarding  the  reward  structure,  "No  good 
deed  shall  go  unpunished."  It's  really  true  that  there's  no  particular  motivation  for  an 
acquisition  manager  to  direct  his  efforts  at  anything  besides  his  own  short-term  goals. 
His  goals  are  fairly  immediate  ones.  By  the  time  the  appointment  of  a  JPA  acquisition 
manager  takes  place,  so  many  degrees  of  freedom  have  been  given  away  that  he  really 
can't  affect  the  long-term  good.  All  he  can  really  do  is  try  to  solve  his  problems  within 
his  budget  and  negotiate  a  way  (whenever  he  can)  to  keep  from  losing  his  shirt  or 
destroying  his  career  (in  the  case  of  a  military  post). 

Ted  Post  argued  in  his  presentation  (as  we  in  the  human  factors  community  have  been 
arguing  for  years)  for  early  involvement  of  JPA  people  in  the  development  process  so  that 
we  can  make  some  kind  of  positive  input  to  design  tradeoffs.  If  we  could  make  known 
some  of  the  things  that  JPAs  have  to  offer  to  those  who  have  power  to  make  design 
tradeoff  decisions,  we  could  possibly  solve  some  of  the  long-term  system  problems  in  a 
cost-beneficial  way.  So  far  this  never  happens,  although  we've  been  pleading  for  it  to 
happen  for  some  time.  We  do  tend  to  get  blamed  for  many  system  shortcomings  of  the 
JPAs  and  the  system  (which  are  not  our  fault)  and  it  doesn't  seem  to  do  much  good  to 
"wring  our  hands"  about  that. 
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We've  talked  about  the  need  for  educating  the  3PA  acquisition  manager  in  terms  of 
aspects  of  3PA  technology.  I  get  a  little  bit  frightened  of  being  dragged  into  the  process 
earlier  when  I  think  about  how  the  government  decides  who  is  going  to  be  involved  in 
solving  the  problem.  We've  said  to  ourselves  that  we  can't  really  know  how  much  a  3PA 
package  is  going  to  cost  unless  we  have  some  grounds  for  scoping  it.  These  grounds  come 
from  analyses  similar  to  those  presented  by  Rosemarie  Preidis.  These  analyses  are  based 
upon  prior  knowledge  of  the  size  and  nature  of  the  system.  That  only  gets  you  a  rough 
estimate  of  the  cost,  however.  Again,  this  is  what  I  thought  the  main  reason  for  our  being 
here  was- -how  the  government  could  be  assisted  in  developing  a  cost  estimate  for  a 
system  and  being  able  to  argue  for  the  appropriate  funds  to  do  the  job.  Without  that  kind 
of  scoping  material  at  an  appropriate  level  of  detail,  how  are  you  going  to  be  able  to 
select  a  contractor?  Maybe  that's  the  wrong  approach,  but  Kay  Inaba  would  argue  that 
there  always  should  have  been  a  several  step  process.  Initially,  there's  a  scoping  effort 
that  decides  how  much  the  product  is  going  to  cost.  Then,  there's  the  initial  development 
of  materials  that  will  serve  as  input  to  the  development  process.  Finally,  there's  the 
practitioner  who  uses  his  "guaranteed  technically  accurate"  material  to  go  on  and  produce 
the  3PAs.  At  that  point,  the  only  difference  among  the  potential  bidders  is  along 
dimensions  that  we've  been  hearing  about  here  from  people  who  are  presently  producing 
3PAs.  These  dimensions  are  the  "technique  stuff"  and  automated  processing,  and  they 
deal  with  many  of  the  production  details.  The  problems  related  to  scoping  in  the  first 
place  are  of  such  an  enormous  magnitude  that  any  one  of  them  probably  adds  up  to  more 
than  the  total  difference  betwen  potential  bidders. 

As  part  of  the  discussion  of  Ted  Post's  presentation,  Kay  Inaba  pointed  out  that 
whoever  uses  LSAR  sheets  complains  that  the  writers  didn't  know  what  they  were  doing 
and  that  they  never  talked  to  one  another.  I'm  less  and  less  convinced  that  the  single 
LSAR  product  is  ever  going  to  be  useful  to  all  the  people  who  are  trying  to  use  it.  The 
training  people  have  needs  that  are  different  from  the  3PA  people,  whose  needs  are 
different  from  the  "engineering  type"  people.  I  think  that's  why  we  see  so  much 
redundancy;  it's  not  just  a  timing  problem.  People  really  do  have  different  qualitative 
needs  that  are  not  satisfied  by  a  single  thing  like  LSAR,  and  that's  why  I  think  we  go  off 
and  do  our  own  analyses. 

Fred  Hart  pointed  out  that,  as  bad  as  training  is,  at  least  objectives  are  systemati¬ 
cally  stated  for  it.  This  is  something  that  is  not  really  systematically  done  for  3PAs.  I 
guess  if  we  could  somehow  sidestep  this  issue  of  cost,  many  of  us  would  argue  that  we 
should  expend  our  efforts  to  figure  out  ways  for  quantifying  performance  that  we  would 
hope  to  get  as  a  result  of  performance  aid  packages.  Then,  we  could  make  an  attempt  to 
buy  some  amount  of  performance  per  dollar  rather  than  some  number  of  pages  per  dollar. 

Some  of  the  kinds  of  questions  that  I  hoped  we  could  look  at  in  more  detail  began  to 
emerge  in  Fred  Hart's  presentation.  It  seems  to  me  that  if  we  look  at  some  of  the 
problems  in  3PA  development  that  derive  from  the  shortcomings  in  the  technical  base 
that  we  are  using,  we  could  back  up  and  rethink  some  of  the  methods  for  the  analyses  and 
the  specifications  that  we  are  using  initially.  For  example,  Fred  mentioned  that  task 
identification  is  relatively  straightforward.  It's  important  that  you  identify  all  of  the 
tasks.  What  do  you  do  if  the  people  that  give  you  this  technical  information  haven't 
identified  all  the  tasks,  or  if  they  have  identified  the  wrong  ones,  or  there  has  been  some 
type  of  mix  up?  It  doesn't  take  many  changes  in  the  number  or  types  of  tasks  or  the 
proportion  of  effort  that  you  have  to  put  into  these  analyses  to  make  a  big  difference  in 
your  cost.  So  what  are  the  implications  of  input  data,  which  are  usually  incomplete  to 
some  degree?  What  if  the  target  audience  doesn't  turn  out  to  be  exactly  like  the  guys 
that  the  government  told  you  that  you  would  have  to  work  with?  We  could  identify  how 
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much  of  an  impact  such  changes  would  have  on  cost  of  developing  a  whole  package  for  a 
user  group  that  was  different  from  the  one  originally  prescribed.  That  would  help  you 
decide  how  much  effort  you  should  put  into  deciding  what  that  user  group  is  going  to  be. 
Back  in  the  days  of  "Vietnamization,"  we  had  anticipated  working  with  people  who  were 
going  to  have  a  year  or  two  of  electronics  experience  and  it  turned  out  that  they  had 
never  seen  anything  more  complicated  than  a  shovel.  That  had  some  profound  implica¬ 
tions  for  the  amount  of  work  you  had  to  put  into  your  package. 

How  does  the  procuring  agency  know  how  much  troubleshooting  coverage  it's  going  to 
get?  Many  people  have  talked  about  that,  but  I  haven't  heard  anything  solid  about  how  we 
would  like  to  see  troubleshooting  coverage  specified.  We  used  to  try  to  flag  every  item  in 
the  task  identification  matrix  for  which  we  felt  that  a  failure  mode  effect  analysis  should 
be  done.  Then,  we  would  ignore  any  failure  of  the  performance  aid  in  finding  a  system 
failure  that  wasn't  specifically  indicated.  This  method  was  utilized  for  fully  procedur- 
alized  JPAs.  I  really  don't  have  any  idea  how  some  of  the  partially  proceduralized  JPAs 
would  fare  in  terms  of  their  ability  to  capture  all,  or  some  part,  of  the  failures  that  occur. 
So,  trying  to  provide  some  sort  of  guidance  to  you — the  JPA  buyer — as  far  as  what 
coverage  and  types  of  troubleshooting  you  want,  is  something  that  should  be  done.  I  don't 
think  we  know  how  to  do  that  right  now. 

How  good  is  the  built-in  test  equipment?  That  topic  slid  by  pretty  quickly  when  it 
was  brought  up  yesterday.  That  has  some  pretty  profound  implications  too.  That  would 
provide  a  real  good  opportunity  for  tradeoff  if  the  JPA  people  were  put  into  the  process 
at  the  point  where  JPA  solutions  and  built-in  test  solutions  could  be  tradedoff.  We  seldom 
get  brought  in  that  early,  though. 

Kay  Inaba  and  several  of  the  rest  of  you  have  indicated  that  you  are  relatively 
comfortable  with  your  ability  to  control  the  costs  for  JPA  generation.  You  are  not 
comfortable  with  the  customer's  ability  either  to  control  or  guarantee  completeness  and 
accuracy  of  the  technical  information  that  comes  out  of  the  front-end  analyses.  I  think 
the  point  about  inadequacy  of  that  technical  information  implies  that  we  have  to  create 
some  type  of  guidance  for  the  analyst-practitioner  as  opposed  to  the  generator- 
practitioner — one  point,  who  is  that  analyst-practitioner  and  who  should  he/she  be? 

Bill  Conroy  showed  us  a  small  scale  example  of  the  problems  and  results  of 
inadequacies  in  the  up-front  data.  In  this  case,  it  was  the  number  of  tasks  that  was  wrong 
and  not  only  that.  When  the  tasks  were  finally  agreed  upon,  it  was  discovered  that  the 
content  of  the  additional  tasks  was  different  from  what  was  originally  planned.  That  kind 
of  thing,  I  feel,  has  much  bigger  cost  implications  than  how  you're  going  to  do  your 
graphics.  The  cost  difference  in  doing  a  single  job  resulting  from  shortcomings  in  the 
technical  base  that  you  have  to  work  with  can  be  much  bigger  than  the  total  cost 
difference  resulting  from  the  internal  guidelines  that  determines  whether  Bill  Conroy, 
with  his  organization's  flexibility,  can  do  a  better  job  at  a  lower  price  than  Fran  Rahl, 
with  his  organization's  relatively  rigid  structure  and  rules. 

Rosemarie  Preidis'  model  seems  to  have  much  more  value  for  the  JPA  buyer  than  for 
the  JPA  developer.  It  seems  to  me  to  have  value  in  coming  up  with  a  ball-park  estimate 
of  the  problem  that  faces  you,  the  JPA  buyer.  The  questions  that  I  have  are:  At  what 
point  should  you  be  trying  to  implement  such  a  model?  When  is  it  reasonable  to  expect 
that  we'll  have  a  chance  to  make  an  estimate  like  that,  talk  to  the  people  who  are  going 
to  control  all  the  system  development  dollars,  and  make  a  pitch  for  a  proportion  of  that 
money?  It  strikes  me  that  we  almost  always — in  spite  of  the  fact  that  we're  talking  about 
JPA  cost  estimation — "back  into  the  process,"  knowing  how  much  overall  money  we  have 
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to  spend.  We  generally  have  some  idea  how  much  money  is  going  to  be  available  for  a 
particular  procurement  and,  although  we  make  much  noise  about  being  very  careful  and 
quantitative  about  doing  our  cost  estimate,  more  often  than  not  we're  "backing  into  it." 

Discussion 


Mr.  3oyce  was  asked  if  he  thought  that  BITE  could  be  offset  by  personnel 
considerations  in  a  tradeoff  situation.  He  replied  that  this  might  happen  if  the  system 
were  understood  to  the  point  of  specifying  what  testing  the  BITE  and  3PA  would  be  doing. 
The  relative  merit  of  BITE  was  controversial  and,  although  the  philosophy  of  BITE  was 
deemed  worthwhile,  the  implementation  showed  that  a  man-in-the-loop  was  still  required 
and  the  possibility  exists  that  BITE  could  very  well  make  a  troubleshooting  job  more 
difficult.  A  point  was  made  that  most  of  the  Navy  equipment  was  not  amenable  to  BITE, 
and  electronic  equipment  only  contributed  to  a  small  percentage  of  the  total. 

Discussion  of  the  acquisition  process  and  problems  with  ILS  continued.  Institutionali¬ 
zation  of  3PA  documentation,  such  as  the  SPA  effort  in  the  Army,  was  agreed  to  be  a 
first  step  in  improvement  of  this  process,  followed  by  delegation  of  responsibility  for 
knowledge  of  3PA  specifications  to  the  system  acquisition  manager  or  a  permanent  staff 
person. 
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COMPARISON  OF  TWO  COST  ESTIMATION  METHODS 


Mr.  John  G.  Bean 
Hughes  Aircraft  Company 
Los  Angeles,  California 


Presentation 


General  Methods  of  Cost  Estimation 


There  are  two  apparently  different  methods  of  estimating  the  cost  of  TI  (job 
performance  aids,  technical  manuals,  training  materials,  etc.):  (1)  total  package  and  (2) 
element. 

Total  Package  Estimation.  In  method  1  (above),  the  cost  of  a  total  package  is 
estimated  by  extrapolating  (on  the  basis  of  a  complexity  ratio  or  factor)  from  the  actual 
cost  data  of  a  similar  package.  For  example,  the  cost  of  the  F-18  program  is  extrapolated 
from  the  actual  cost  of  the  F-14  program;  the  cost  for  an  operator  manual  for  the  F-18 
aircraft  is  extrapolated  from  the  historical  cost  for  an  operator  manual  for  the  F-14 
program,  etc.  (see  Slide  1). 

Element  Estimation.  In  method  2  (above),  actual  cost  of  an  individual  element  within 
the  total  package  is  utilized  and  all  the  various  costs  of  individual  elements  are  added  up 
to  derive  the  total  cost.  For  example,  the  cost  of  a  schematic  times  the  number  of 
schematics,  plus  the  cost  of  an  isometric  drawing  times  the  number  of  isometric  drawings, 
plus  the  cost  of  a  table  of  text  times  the  number  of  tables  of  text,  and  so  forth  for  all  the 
elements  in  the  package  (see  Slides  2  and  3). 

Convergence  of  Methods  of  Cost  Estimation.  The  total  package  and  element  methods 
should  lead  to  identical  results  when  properly  applied.  Any  total  package  can  be  broken 
down  into  its  constituent  elements  (schematics,  functional  diagrams,  analysis,  text,  parts 
lists,  validation,  edit,  etc.)  and  the  cost  of  a  total  package  prorated  among  the  constituent 
elements  based  on  a  historical  cost  data  base  or  "engineering  judgement." 

The  following  paragraphs  discuss  some  of  the  key  parameters  that  can  affect  the  TI 
cost  estimating  process. 

Cost  Estimation  Accuracy 

Cost  estimation  accuracy  boils  down  to: 

1.  The  accuracy  of  the  comparison  (engineering  judgment)  of  a  new  package  to  be 
estimated  to  a  package  for  which  actual  cost  information  is  available  either  on  a  total 
package  or  elemental  basis. 

2.  The  accuracy  of  the  actual  cost  information  that  has  been  collected. 

Accuracy  of  Comparing  New  With  Previously  Developed  Package/Element.  Accurate 
cost  estimation  is  easy  if  the  task  to  be  estimated  and  accomplished  is  very  similar  in 
structure  and  nature  to  a  task  already  accomplished  for  which  accurate  historical  cost 
data  are  available.  If  an  F-18  flight  manual  is  deemed  to  be  about  the  same  complexity  as 
an  F-14  flight  manual  and  the  F-14  flight  manual  cost  was  $150K,  then  the  F-18  flight 
manual  should  also  cost  about  $150K  (providing  it  is  to  be  developed  by  the  same 
contractor  in  a  very  similar  working  and  cost  environment). 
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Technical  Information.  (TI)  Cost  Estimation.  Model 


N  =  Cost  of  new  TI  package  (see  Slide  3  for  package  example)  being  estimated 

S  =  Actual  cost  of  similar  TI  package  previously  developed  (from  historical 
data) 

F  =  Complexity  factor  (based  on  engineering  judgement) 

E  =  Cost  of  TI  element  (Slide  2)  in  previously  developed  TI  package 


N  =  FS  (Total  Package  Estimation  Method) 


or 


n 

N  =  I  F.E.  (Element  Estimation  Method) 
.  .  11 

i=l 


n 

FS  =  1  F_^E_^  (Convergence  of  Methods) 


F  (Complexity  Factor)  is  based  on  such  considerations  as: 


(a)  No.  of:  LRU’s  Subsystems 

SRA’s  Modules 

Systems  Assemblies 

Functions  Test  Points 

Tasks 

(b)  Increased  or  decreased  capability 

(c)  Volume 

(d)  Weight 

(e)  Amount  of  Built-in  Test  (BIT) 

(f)  Amount  of  Automatic  Test  Equipment  (ATE) 

(g)  Amount  of  Equipment  Changes 

(h)  Actual  count  of  No.  of  changed  or  affected  pages 

(i)  Percent  of  prime  equipment  cost 

(j)  Equipment  type  (electronic,  mechanical) 


Bean:  Slide  1 
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Examples 

of  TI  Cost  Elements  (1  of  2) 

Any  complete  set  may  be  selected. 

A  complete  set  sums  to  the  total  TI  package.  Example  of  typical  complete 

set:  Writing,  illustrating, 

typing,  other. 

*  Writing 

*  Typing 

Analysis 

Page  Layout 

*  Task 

Composition 

*  Theory 

Proofing 

*  Troubleshooting 

Rework 

Research 

Supervision 

Planning 

Liaison 

Source  Data 

Other  Considerations 

Gathering 

*  Conversion 

Level  of  Detail 

Editing 

Maintenance  Concept  (throw-away 

Checking 

versus  detailed  repair) 

*  Validation 

No.  of  LRUs 

Parts  Listing 

No.  of  SRAs 

*  Inspection 

No.  of  Major  Systems,  Subsystems 

Rework 

Units,  Assemblies,  or  Modules 

*  Verification 

Schedule 

Supervision 

Page  Size 

Information  Density 

Program  Phase  (Research, 

Other  Costs 

Development,  Test,  Prototype, 
Production) 

Materials 

*  Customer  Direction 

Travel 

*  Equipment  Availability 

Printing 

Copies 

*  Illustrating 

Photography 

Layout 

Inking 

*  Cost  element  which  contributes  signifi¬ 

Checking 

cantly  to  TI  cost  and  which  can  be 

Rework 

reduced  through  improved  methodology. 

Half-tone 

Exploded  View 

Line 

Cutaway 

Mechanical 

Supervision 

Bean:  Slide  2 
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Examples  of  TI  Cost  Elements  (2  of  2) 


Types  of  Information 

Title  Page 

Warning  Page 

List  of  Effective  Pages 

Table  of  Contents 

Introduction 

Federal  Mfr  Codes  and  Addresses 

Parts  List 

Numerical  Index 

Reference  Designator  Index 

Alphabetical  Index 

Text 

Description 

theory 

^Diagrams 

Schematic 

Functional 

Test 

Troubleshooting 

Procedures 

Operator 

Mechanical 

Test 

troubleshooting 
*Logic  Trees 

Maintenance  Dependency  Charts  (MDC) 
Program  Listings 
Wire  Lists 
Check  Lists 
Lists 
Tools 

Test  Equipment 
Materials 


*  Cost  element  which  contributes  significantly  to  TI  cost  and  which  can 
be  reduced  through  improved  methodology. 


Bean:  Slide  2  (Continued) 
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Examples  of  TI  Packages 


Flight  Manuals 
Maintenance  Manuals 
Organizational 
Intermediate 
Depot 

Operator  Manuals 

Job  Performance  Aids  (JPA) 

Fully  Proceduralized  Job  Performance  Aids  (FPJPA) 
Maintenance  Requirement  Cards  (MRCs) 

Parts  Lists 

Illustrated  Parts  Breakdowns  (IPB) 

Student  Learning  Guides 
Instructor  Guides 
Computer  Program  Listings 
Wire  List  Manuals 
Troubleshooting  Manuals 
Checklists 
Publication  Indexes 
Tape  Indexes 
Diagram  Manuals 
Work  Package 
Job  Guide 


Bean:  Slide  3 


The  cost  estimation  situation  is  much  more  complex  if  the  previous  and  new  packages 
are  developed  by  different  contractors.  Here  it's  necessary  to  consider  the  effect  of 
differences  in  extraneous  factors  such  as  overhead  rate,  labor  grade  structure,  standard 
operating  procedures,  etc.  For  example,  one  contractor  may  do  all  the  writing  on 
overhead  and  charge  directly  only  for  the  artwork  and  typing;  another  contractor  may  do 
just  the  opposite.  One  contractor  may  charge  for  validation;  another  may  include  it  as 
part  of  the  hardware  cost,  etc.  These  inherent  differences  between  contractors  result  in 
contractors  having  substantially  different  prices  for  what  appears  to  be  a  similar  task. 

Historical  cost  information  thus  is  valid  only  when  applied  to  very  similar  circum¬ 
stances  for  the  same  TI  developer.  This  restriction  doestnot  apply  to  procurements  in 
which  the  effects  of  extraneous  factors  are  neutralized.  For  a  competitive  procurement 
of  a  stand-alone  package,  the  total  cost  should  be  comparable  even  though  the  package  is 
estimated  by  different  developers  and  the  elemental  costs  vary  widely. 

In  addition,  for  the  estimate  to  have  any  real  validity,  it  must  have  been  developed  by 
relating  it  to  a  similar  job.  Whenever  new  work  procedures  are  instituted  or  new  products 
are  produced,  then  the  initial  development  may  take  much  more  effort  than  the  steady- 
state  repetitive  development.  (A  good  rule  of  thumb  is  10:1.)  If  an  organization  has  never 
prepared  a  flight  manual  before,  their  first  effort  may  take  them  many  times  more  effort 
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than  subsequent  flight  manual  efforts.  Their  estimate  for  their  first  attempt  would  be 
suspect;  subsequent  estimates  based  on  their  first  attempt  would  also  be  suspect  and 
probably  substantially  higher  than  that  at  the  end  of  the  learning  process. 

Accuracy  of  Summing  Cost  of  Individual  Elements  to  Derive  Cost  of  Total  Package. 

Many  organizations  believe  that,  somehow,  with  great  masses  of  estimating  detail,  they 

will  end  up  with  accurate  estimates.  This  is  analogous  to  believing  wrongly  that,  if  the 
value  of  something  is  known  to  the  nearest  percent  and  if  all  arithmetic  is  carried  out  to 
a  thousandth  of  a  percent,  the  final  result  will  be  much  more  accurate  than  a  percent. 

In  many  cases,  the  converse  is  true  because  so  much  time  and  effort  are  spent 
worrying  about  the  details  that  the  critical  factors  become  lost  in  the  exercise.  A 
careful  and  assiduous  program  of  identifying  the  cost  elements  and  carefully  pricing  each 
element  out  can  lead  to  accurate  estimates.  But  when  the  level  of  detail  necessary  to 
support  the  element  estimation  method  is  not  available  or  is  greatly  inaccurate,  the  mass 
of  backup  estimating  data  produced  is  of  little  value  and,  as  indicated  above,  critical 
factors  are  often  overlooked  in  the  detail.  For  example,  in  estimating  the  cost  of  a  flight 
manual,  the  cost  of  the  command  (NATOPS)  reviews  might  be  overlooked  because  the 
estimator  overlooked  that  a  NATOPS  conference  was  required.  The  total  package 
method  would  automatically  include  this  cost. 

Key  Cost  Factors  In  TI  Development 

The  following  paragraphs  provide  a  discussion  of  some  of  the  key  cost  factors  in 
development  of  TI.  The  factors  discussed  are  felt  to  be  those  that  greatly  affect  the  cost 
of  TI. 

Direct  Pickup  of  Source  Data.  The  less  writing,  illustrating,  typing,  and  editing  that 
has  to  be  accomplished,  the  less  TI  development  will  cost.  Conversely,  the  more  rework, 
reformatting,  redrawing,  rewriting,  and  originating  that  has  to  be  done,  the  more  TI 
development  will  cost.  A  key  factor  in  the  cost  of  TI  development  is  the  amount  of 
material  that  can  be  directly  picked  up  and  the  amount  that  must  have  substantial 
rewriting  or  original  writing  before  it  can  be  used. 

One  way  to  reduce  the  cost  of  TI  greatly  is  to  pick  it  up  directly  whenever  possible. 
The  government  should  insist  that  their  equipment  contracts  require  that  the  engineering 
drawings,  manufacturing  and  test  procedures,  specifications,  etc.  be  developed  in  such  a 
way  that  they  can  be  used  directly  in  TI.  The  schematic  diagrams  in  TI  should  be  direct 
pickup  engineering  drawings  and  not  be  redrawn  or  relaid  out.  Similarily,  wire  lists, 
program  lists,  parts  lists,  and  test  procedures  should  be  prepared  originally  for  direct 
pickup  in  TI.  This  approach  not  only  greatly  reduces  the  cost  of  TI,  but  it  also  greatly 
improves  quality  by  eliminating  the  many  transcription  errors  that  are  made  in  repro¬ 
cessing  source  material. 

Level  of  Detail.  TI  must  be  written  to  the  level  of  the  intended  user,  eliminating  all 
unnecessary  detail  above  or  below  his  level,  but  provide  enough  information  so  the  user 
can  do  his  job  without  error. 

1.  If  the  user  knows  how  to  use  a  tool  or  item  of  test  equipment,  then  procedures 
merely  need  words  to  the  effect,  "Using  (tool  or  test  equipment  item)  .  .  .  ." 

2.  If  the  user  does  not  know  how  to  use  the  tool  or  test  equipment  item,  detailed 
step-by-step  procedural  information  is  needed  either  in  the  specific  procedure  or  by 
reference  to  another  procedure. 
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3.  If  the  user  has  developed  a  skill  or  learned  the  knowledge  in  his  training  course, 
then  explicit  information  should  not  be  given  but  should  be  assumed  to  be  known  by  the 
user. 


4.  If  the  user  has  not  mastered  the  skill  or  cannot  recall  knowledge  from  memory 
(i.e.,  as  the  result  of  a  training  course),  step-by-step  instructions  or  detailed  information 
will  be  necessary. 

During  the  writing  process,  the  minimum  amount  of  detail  thought  necessary  should 
be  developed.  If,  at  validation,  it  is  found  that  the  information  is  not  adequate  for  the 
user  to  do  the  task  the  first  time  without  error,  then  additional  information  can  be 
provided  and  revalidated. 

One  of  the  functions  of  job  performance  TI  is  to  provide  a  reference  document  where 
the  user  can  go  if  he  has  questions  about  his  procedure.  So,  even  for  a  task  where  the  user 
would  normally  be  expected  to  know  all  the  detailed  steps,  it  still  may  be  necessary  to 
provide  the  detailed  reference  procedure  in  the  job  performance  document.  The  ultimate 
test  of  the  level-of-detail  match  is  a  demonstration  that  a  user  with  aptitude,  training, 
and  knowledge  equivalent  to  that  of  the  intended  user  can  actually  perform  the  necessary 
tasks  without  error. 

Thus,  there  is  a  tradeoff  between  TI  and  training.  With  additional  training,  the  user 
will  need  less  information  in  his  TI  documents  and  vice  versa.  In  general,  this  tradeoff 
must  be  accomplished  for  each  individual  program.  Training  is  a  recurring  cost  and  TI  is  a 
nonrecurring  cost.  Where  large  numbers  of  technicians  are  encountered,  it  may  be 
cheaper  to  include  the  necessary  information  in  the  job  documents,  rather  than  to  provide 
an  extensive  training  program.  On  the  other  hand,  with  limited  quantities  of  equipment 
and  small  numbers  of  technicians,  then  an  extensive  training  program  may  be  the  most 
economical  approach. 

Analysis.  One  of  the  highest  cost  factors  in  TI  development  is  engineering  or  analysis 
effort.  This  includes  task  analysis,  theory-of-operation  preparation  analysis,  and  trouble- 
shooting-procedure  development  analysis. 

1.  Task  analysis.  Task  analysis  is  becoming  increasingly  important  in  TI  develop¬ 
ment.  Prior  to  the  presentation-of-information-for-maintenance-organization  (PIMO) 
program,  task  analysis  was  given  little  emphasis  by  TI  developers.  Properly  carried  out 
task  analysis  leads  to  better  TI,  developed  at  lower  cost  because  the  proper  tasks  are 
included  in  the  technical  document  with  the  right  level  of  detail;  extensive  rework  and 
modification  to  compensate  for  a  poor  task  analysis  are  unnecessary. 

Potentially,  logistic  support  analysis  records  (LSARs)  are  a  valuable  task  analysis 
information  source  for  TI  generators.  However,  the  process  by  which  LSARs  are  currently 
developed  and  documented  is  not  TI  oriented.  Shortcomings  in  the  LSARs  seriously  impair 
their  value  for  TI  task  analysis.  As  a  result,  in  many  cases,  a  separate  duplicative  analysis 
is  carried  out  specifically  for  TI  generation.  A  concerted  effort  should  be  made  to 
integrate  the  LSAR  analysis  outputs  for  direct  use  as  the  task  analysis  baseline  for  both 
the  training  and  TI  development  process. 

2.  Theory-of-operation  analysis.  Theory  of  operation  has  the  objective  of  support¬ 
ing  troubleshooting  by  providing  information  that  enables  the  technician  to  understand  the 
equipment  and  figure  out  (from  his  understanding)  how  to  fix  it.  Ordinarily,  theory  of 
operation  is  not  presented  in  a  job  context  where  it  is  related  to  a  specific  job  task  but, 
rather,  as  general  information  somehow  applicable  to  all  tasks. 
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Since  a  highly  skilled,  subject  matter  expert  is  required  to  write  the  theory  of 
operation,  its  cost  is  usually  high.  Because  personnel  of  this  capability  are  in  short  supply, 
usually  the  theory  material  is  inaccurate  and  incomplete. 

Conventional  theory  formats  usually  provide  little  information  except  the 
obvious:  Signal  x  is  applied  to  component  x,  which  generates  a  new  signal  y  and  applies  it 
to  component  y,  etc.  In  most  cases,  the  theory  of  operation  is  extremely  difficult  to  read 
and  understand  and  adds  significant  volume  to  the  technical  document. 

In  the  conventional  approach,  textual  material  is  intended  to  convey  all  of  the 
essential  information.  Several  new  methods  are  available  that  greatly  improve  theory  of 
operation,  reduce  the  cost  of  its  development,  and  reduce  its  volume.  These  techniques 
use  the  block,  functional,  or  schematic  diagram  as  a  basis  and  then  key  the  textual 
material  specifically  to  the  diagram. 

3.  Troubleshooting  procedure  development  analysis.  Troubleshooting  procedures 
are  the  most  difficult  part  of  TI  to  develop  and  are,  thus,  very  expensive.  Computer- 
developed  troubleshooting  promises  to  provide  a  means  to  reduce  significantly  the  cost  of 
preparation  of  troubleshooting  information  and  also  provide  much  higher  quality  trouble¬ 
shooting  information. 

Computer-developed  troubleshooting  enters  a  model  of  equipment  in  a  computer 
data  base  and  then  allows  the  technician  to  input  test  result  information.  A  computer 
program  then  processes  the  test  results,  determines  the  next  test  for  the  technician  to 
make  and  enter  results  about,  and  so  on,  until  the  trouble  has  been  isolated.  This  type  of 
troubleshooting  does  not  rely  on  technician  understanding  of,  or  knowledge  about,  the 
equipment  in  order  to  fault  isolate,  so  requirements  for  theory  of  operation  and  other 
troubleshooting  type  of  information  in  the  TI  document  are  minimized. 

Computer-aided  Authoring.  Substantial  reductions  in  TI  development  costs  can  also 
be  achieved  by  the  application  of  computer-assisted  authoring.  Computer-assisted 
authoring  prompts  a  subject  matter  expert  (who  is  usually  not  an  expert  writer  or 
instructional  technologist)  to  input  source  information  that  is  then  automatically  for¬ 
matted  and  processed  into  the  final  TI  document.  Utilizing  computer  data  bases  for 
vocabulary,  spelling,  tools,  nomenclature,  etc.  can  greatly  reduce  the  burden  on  the 
writers  and  editors,  the  inconsistencies  within  and  between  technical  documents,  and 
greatly  increase  TI  writer  productivity. 

Illustrations.  Another  high  cost  item  in  TI  development  is  the  preparation  of 
illustrations.  One  reason  for  the  high  cost  of  graphics  is  overspecification  of  require¬ 
ments.  These  overspecifications  include  the  following: 

1.  Requiring  too  much  detail;  overelaborateness. 

2.  Requiring  too  much  fidelity. 

3.  Overemphasis  on  angle  of  view. 

Computer  graphics  promises  to  reduce  cost  of  illustration  development  greatly  by 
increasing  productivity  of  artists.  It  still  will  be  important  to  review  the  contractual 
specifications  to  assure  that  unnecessary  requirements  are  not  imposed  and  that  illustra¬ 
tions  contain  only  the  bare  minumum  of  detail  necessary  to  portray  the  intended 
information  clearly. 
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Source  Data  Accuracy.  Accuracy  and  availability  of  source  data  (such  as  engineering 
drawings,  test  specifications,  test  procedures,  etc.)  greatly  affect  TI  cost.  The  following 
are  typical  source  data  problems: 

1.  Inaccurate  or  incomplete. 

2.  Continual  change. 

3.  Late  release  or  incorporation  of  changes. 

When  the  engineering  source  data  are  not  available  or  rapidly  changing,  the  Tl  writing 
process  is  in  a  constant  state  of  change  and  confusion.  In  some  programs,  this  is 
necessary  due  to  the  high  priority  of  getting  the  equipment  out  to  the  field  as  rapidly  as 
possible.  However,  in  most  programs,  adequate  planning  can  minimize  the  impact  of  this 
change  process.  It  may  be  well  in  rapidly  changing  programs  to  recognize  the  situation 
and  turn  out  the  document  in  an  evolutionary  manner.  (For  example,  deliver  the  essential 
procedural  information,  the  schematic  and  functional  diagrams,  the  troubleshooting 
procedural  data,  and  so  on.) 

Customer  Direction.  Changes  in  direction  by  the  Tl  customer  can  be  another  major 
cost  factor.  In  some  cases,  this  comes  about  because  the  contracting  officer's  represen¬ 
tative  is  reassigned  in  the  middle  of  a  program  causing  a  new  representative  to  be 
assigned  who  then  brings  his  own  viewpoints  to  the  project.  In  other  cases,  it  is  simply  a 
matter  of  subjective  change  of  direction  that  causes  extensive  rework  and  high  associated 
costs.  In  many  cases,  agreeing  on  a  baseline  sample  early  in  a  program  or  including  a 
sample  of  the  desired  product  with  the  request  for  proposal  will  greatly  clarify  customer 
requirements  and  minimize  changes  in  direction.  The  following  are  some  examples: 

1.  Change  of  mind. 

2.  Inconsistent  direction. 

3.  Subjective  comments. 

Equipment  Availability.  When  the  equipment  is  available,  the  TI  preparation  task  is 
greatly  simplified  and,  consequently,  costs  can  be  greatly  reduced.  Instead  of  the  writer 
or  artist  having  to  guess  about  an  aspect  of  the  equipment  or  trying  to  figure  it  out  from 
the  engineering  drawings,  he  can  get  up  and  go  over  and  look  at  the  actual  equipment, 
take  a  photograph,  make  a  sketch,  etc. 

Similarly,  in  the  validation  process,  it  is  essential  that  hardware  be  made  available  to 
support  the  test  of  the  TI  validity.  If  the  equipment  is  not  available  for  validation,  the 
procedural  information  will  just  not  be  accurate. 

Data  Entry.  The  conventional  way  to  develop  TI  is  for  the  writer  to  prepare  a  draft 
of  the  material  in  long  hand,  give  it  to  a  typist  who  prepares  a  typed  draft  and  returns  it 
for  review,  make  corrections,  and  send  it  back  to  the  typist  for  correction.  This  process 
continues  until  the  document  is  finalized. 

This  process  is  both  time-consuming  and  costly.  Many  TI  developers  soon  will  greatly 
increase  the  productivity  of  the  data  entry  process  by  placing  data  entry  terminals  in  the 
hands  of  the  technical  writers  who  then  enter  the  data,  make  the  necessary  revisions  and 
corrections,  and  print  out  the  final  copy  without  ever  utilizing  the  services  of  typing  or 
production  personnel.  It  is  also  expected  that  many  TI  developers  will  combine  this 
process  with  computer-developed  graphics  so  that  the  entire  Tl  document  production 
process  is  essentially  automated. 
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Problems  Associated  With  Utilizing  Cost  Per  Page 


In  preparing  and  evaluating  proposals,  contractors  and  the  customer  (government) 
need  easily  identified  elements  against  which  to  associate  cost.  Pages,  frames, 
illustrations,  data  modules,  etc.  are  examples  of  such  units.  The  existence  of  much  valid 
historical  cost  data  based  upon  these  units  further  strengthens  the  argument  for  their 
continued  use  in  pricing. 

There  are  problems  associated  with  the  use  of  these  units  for  pricing,  however.  On 
one  hand,  the  contractor,  in  order  to  maximize  his  profit,  desires  to  bid  the  maximum 
number  of  units  against  which  cost  can  be  legitimately  associated  at  the  maximum  cost 
per  unit.  On  the  other  hand,  the  government,  in  an  attempt  to  keep  program  costs  as  low 
as  possible,  strives  to  keep  the  total  number  of  units  and  the  TI  unit  cost  to  a  minimum. 

In  many  cases,  unfortunately,  the  result  of  this  interplay  is  too  much  emphasis  on  the 
unit  cost  and  not  enough  on  the  total  number  of  units.  In  order  to  actually  reduce  the 
number  of  units,  considerable  effort  is  required  on  the  part  of  the  contractor  to  edit, 
relayout,  and  otherwise  condense  the  information.  When  the  government  is  overly 
sensitive  to  a  cost  per  unit,  the  contractor  is  reluctant  to  take  on  the  extra  effort  that,  in 
fact,  will  increase  his  cost  per  unit  without  compensation.  In  fact,  he  is  penalized 
because  of  his  higher  cost  per  unit  than  less  consciencious  contractors  and  he  may  be  held 
up  to  be  a  "high  per-page-cost"  contractor. 

To  alleviate  these  problems,  both  the  contractor  and  the  government  must  adopt  the 
attitude  that  TI  units  are  only  a  means  of  determining  a  price  for  level  of  effort 
associated  with  production  of  a  TI  package.  The  government  should  not  buy  a  definite 
number  of  units  but,  rather,  an  effort  that  will  produce  the  minimum  number  of  units 
needed  to  accomplish  the  purpose  of  the  TI.  The  contractor  must  be  adequately 
compensated  for  cost  incurred  for  the  additional  effort  associated  with  the  reduction  of 
TI  units,  and  a  higher  cost-per-unit  should  be  expected  and  encouraged  when  the  total 
number  of  units  is  reduced. 

Recom  mendations 


1.  Direct  pickup  of  TI  source  data.  Drawings  and  other  source  data  should  be 
formatted  initially  so  that  they  can  be  used  directly  as  TI. 

2.  Logistic  support  analysis  records  (LSARs).  LSARs  should  be  prepared  in  formats 
that  can  be  directly  used  for  TI  task  analysis. 

3.  Computer-assisted  troubleshooting  (CAT)  development.  CAT  development  and 
use  should  be  encouraged  to  reduce  TI  development  costs  and  improve  the  quality  of 
troubleshooting  information  presented  to  the  user. 

4.  TI  volume  reduction.  A  specific  contractual  obligation  (task)  should  be  required. 
The  contractor  should  have  a  contract  task  to  review  and  boil  down  the  information  to  the 
minimum  essential  TI.  This  task  should  be  funded  and  adequate  time  allotted  for  its 
completion. 

5.  Computer-assisted  authoring  (CAA).  CAA  should  be  encouraged  as  a  way  to 
improve  author  productivity  and  TI  quality. 
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6.  Sample  of  desired  product.  The  customer  should  include  a  sample  of  the  desired 
product  as  part  of  the  request  for  proposal.  This  sample  will  greatly  clarify  how  the 
customer  will  interpret  the  TI  specifications  and  facilitate  accurate  cost  estimating. 

7.  Equipment  availability.  Programs  must  be  more  carefully  planned  so  that 
hardware/ equipment  will  be  available  for  TI  writing  and  validation.  This  may  result  in 
equipment  stretchouts  or  the  requirement  to  purchase  additional  equipment  early  in  a 
program. 

Discussion 


Initial  discussion  of  Mr.  Bean's  models  focused  on  the  models'  utility  and  identifica¬ 
tion  of  a  point  at  which  the  difference  between  old  and  new  systems  made  historical 
information  nonappli cable.  The  group  agreed  that  it  would  also  be  nice  to  have 
operational  and  maintenance  data  on  the  historical  system.  Discussion  of  differences 
between  contractor's  costing  methods  brought  out  the  difficulties  in  comparing  contractor 
costs.  The  problem  of  the  "submersion"  of  technical  manual  costs  within  hardware  and 
total  system  costs  was  discussed.  Mr.  Bean  mentioned  one  company  that  includes  the 
entire  cost  of  technical  manuals  in  their  overhead.  Although  discrepancies  were  noted 
between  companiesj  Mr.  Bean  noted  that,  to  some  degree,  comparison  between  companies 
was  possible,  but  the  comparer  must  be  aware  of  company-peculiar  costing  methods. 

Discussion  then  centered  on  the  cost  and  time  savings  attributable  to  computer-aided 
authoring,  illustrating,  text  editing,  etc.  A  point  was  made  that  fewer  errors  occurred  in 
the  text  when  computer  aiding  was  used,  with  resulting  reductions  in  validation  time  and 
easier  editing/revision  of  revised  materials. 

Discussion  also  covered  the  level  of  detail  necessary  for  the  task  analysis.  Mr.  Bean 
stated  that  the  detailing  of  steps  within  each  task  was  not  necessary  and  that  the 
information  for  task  analysis  should  come  out  of  LSA,  with  the  exception  of  specific  steps 
involved  in  each  task. 


77 


ANALYSIS,  CONCLUSIONS,  AND  RECOMMENDATIONS 


Dr.  Glenn  A.  Osga 
Systems  Exploration,  Incorporated 
San  Diego,  California 

The  major  objectives  of  the  3PA  cost  factors  meeting  were  to  identify  factors  that 
influence  the  cost  of  job  performance  aids,  present  current  methodologies  for  cost 
estimation,  and  develop  preliminary  cost  modeling  ideas  with  some  direction  for  future 
research.  This  information  would  be  used  by  NAVPERSRANDCEN  to  highlight  areas  that 
might  be  defined  further  and  assigned  some  priority  for  research. 

Certain  limitations  on  the  scope  of  the  meeting  were  defined  early.  The  technical 
information  type  was  limited  to  job  performance  aids,  and  this  was  further  restricted  to  a 
paper-domain.  Booher's  (1977)  statement  that  "there  is  no  generally  accepted  definition 
for  a  job  performance  aid — even  among  those  who  have  done  considerable  research  and 
development  in  the  field"  was  "revalidated"  during  this  meeting.  Initial  controversy  over 
the  weighting  of  3PA  cost  versus  3PA  effectiveness  cast  some  doubt  about  attaining  this 
conference  objective.  Fortunately,  despite  differing  philosophies  concerning  the 
definition  of  JPAs  and  the  importance  of  cost  estimation,  some  valuable  information  was 
shared  by  conference  participants. 

The  presentations  can  be  sorted  into  two  groups:  (1)  Those  concerned  with  cost 
estimation  models  (quantitative),  and  (2)  those  concerned  with  identification  of  JPA  cost 
drivers  (qualitative).  Two  presentations  emphasized  information  of  the  first  type;  all 
presentations  contained  some  information  of  the  second  type. 

3PA  Cost  Estimation  Models 


Ms.  Preidis  presented  a  specific  methodology  and  Mr.  Bean  discussed  two  types  of 
cost  estimation  methods.  Ms.  Preidis'  model  predicted  page  numbers  for  12  different  page 
types  using  system  characteristics  as  cost  factors.  The  cost  of  producing  each  page  type 
is  estimated  and  these  elements  are  combined  to  yield  a  total  cost  estimate.  Her  method 
corresponds  to  Mr.  Bean's  "element  cost  estimation"  method.  Another  method  described 
by  Mr.  Bean  was  the  "total  package  estimation"  method  by  which  historical  cost  data  from 
a  similar  system  are  subjectively  "inflated"  to  obtain  an  estimate  for  current  systems. 
The  general  group  consensus  was  that  these  models  are  good  for  rough,  but  not  accurate, 
estimates.  Indeed,  the  remaining  factors  not  contained  in  these  models  are  numerous  and 
substantially  contribute  to  final  cost.  Thus,  for  systems  with  some  historical  precedent, 
with  similar  preproduction  factors,  these  models  may  offer  utility  to  the  procurer.  It 
seems  that  these  models  will  be  restricted  to  production  cost  estimates,  in  terms  of 
accuracy,  given  the  variability  and  impact  of  front-end  costs. 

Mr.  Bean  suggests  that  a  limited  number  of  cost-per-page  categories  be  chosen  for  a 
select  group  of  document  types.  An  example  of  a  standard  for  a  hybrid  3PA  (double¬ 
column,  single-spaced,  10  percent  foldout  pages  of  3  units  average)  might  be  as  follows: 
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Hybrid  3PA 


Cost  Category 

Cost  per  Page 

a. 

Writing 

7  hrs 

b. 

Illustrating 

5  hrs 

c. 

T  yping/Production 

2  hrs 

d. 

Inspection 

1  hr 

e. 

Validation 

2  hrs 

f. 

Material 

$5.00 

Total  cost  per  page 

17  hrs  $5.00 

Mr.  Bean  stated  that  there  would  be  wide  but  not  complete  application  for  such  standards. 
Contractors  would  have  to  establish  fairly  elaborate  accounting  systems  in  the  cost-per- 
page  categories  in  which  to  collect  cost  data  for  each  document  type.  Mr.  Bean 
considered  it  possible  to  do  this  for  a  simple  system,  as  shown  in  the  example.  For  a 
system  such  as  the  one  described  by  Ms.  Preidis,  he  considered  the  cost  accounting  system 
to  be  impractical  at  this  level  of  detail. 

Thus,  it  appears  that  the  cost  modeling  techniques  described  by  participants  may 
have  some  utility  for  production  cost  estimation.  The  accuracy  of  such  methods  will 
depend  upon  the  stability  of  the  front-end  data  preparation  process.  Until  the  front-end 
process  can  be  stabilized,  cost  estimates  for  overall  project  cost  will  remain  a  "hit-or- 
miss"  proposition.  Each  TI  preparation  project,  even  those  for  the  same  document  type, 
will  remain  an  entity  unto  itself,  influenced  by  unique  combinations  of  unaccounted  cost 
factors  that  Eire  not  comparable  to  a  current  effort  because  the  producer  and  procurer 
cannot  be  certain  which  random  variations  in  these  factors  will  affect  current  costs. 

Identification  of  3PA  Cost  Drivers 


Qualitative  information  presented  during  the  meeting  covered  various  subsystems 
within  the  3PA  preparation  system.  Preparation  system  is  used  to  connote  the  entire  life 
cycle  of  a  3PA,  from  the  initial  technical  information  requirement  to  the  final  delivery. 
Figure  1  has  been  prepared  to  show  a  general  3PA  life  cycle  in  which  major  cost  factors 
can  be  subdivided  and  placed.  Six  major  development  phases  are  shown  with  three 
participant  groups.  Cost  milestones  include  the  weighing  of  budget  constraints  versus 
system  needs,  initial  cost  estimates,  budget  definition,  and  final  cost  before  and  after  a 
product-fix  phase.  Further  costs  may  be  involved  when  system  updates  necessitate  3PA 
data  updates. 

A  major  point  emphasized  during  the  meeting  was  the  discrepancy  between  the  ideal 
and  usual  budget  development  process.  The  ideal  process  involves  the  definition  of  user 
needs  and  specification  of  proper  TI  to  meet  those  needs.  Budgeting  then  involves  the 
fitting  of  a  cost  estimate  to  the  data  collection  process  and  TI  preparation  (i.e., 
effectiveness  first,  cost  second).  The  usual  budget  development  sequence  followed  is  (1) 
allocate  budget,  (2)  define  what  can  be  done  for  the  allocated  dollars,  and  (3)  try  to  make 
that  meet  user  needs  (i.e.,  cost  first,  effectiveness  second).  Although  Figure  1  is  an 
accurate  reflection  of  the  preparation  process  as  described  in  various  DoD  documents,  it 
represents  the  ideal  rather  than  the  usual,  probably  because  the  usual  looks  even  worse 
when  shown  in  print. 


SO 


81 


Figure  1.  Job  performance  aid  preparation  process  including  competive  bid/producer  collection  of 
input  data. 


Figure  1  is  prepared  to  allow  for  a  system  framework  in  which  the  cost  factors  can  be 
contained;  however,  one  must  recognize  that  all  the  variations  of  this  process  cannot  be 
shown  and  must  be  incorporated  by  the  reader.  Points  of  difference  may  include 
situations  where  there  is  a  noncompetitive  bid,  input  data  are  prepared  by  the  procurer, 
input  data  are  fixed  by  the  preparer,  validation  is  done  by  the  procurer,  and  no  product-fix 
phase  exists. 

While  many  cost  factors  were  identified,  conference  attendees  focused  on  those 
related  to  cost  milestones  and  to  the  participant  characteristics  that  affect  cost.  The 
various  cost  factors  were  subsequently  grouped  into  one  of  seven  areas,  identified  in 
Table  1.  Each  area  was  subsequently  broken  down  into  the  individual  factors.  These 
factors  and  attendees  who  stressed  each  are  presented  in  Table  1. 

The  number  of  dots  in  the  Table  1  matrix  should  not  be  evaluated  as  indicating  factor 
importance,  as  each  participant  chose  to  focus  his  or  her  attention  on  different 
development  phases  and  factors.  Table  1  also  contains  cost-reduction  suggestions  voiced 
during  the  meeting  and  items  that  drive  up  cost  or  increase  variability  in  cost  estimates. 
Each  cost-factor  area  is  discussed  separately.  Interactions  between  factors,  however, 
must  be  recognized  as  they  may  create  other  cost  problems. 

Cost  Factor  Areas 


Personnel/Equipment/Environment  Characteristics 

These  factors  include  physical  characteristics  and  attributes  of  the  personnel, 
equipment,  and  environmental  conditions  in  which  they  operate.  Equipment  character¬ 
istics  are  more  numerous  than  those  shown  in  Table  1,  but  these  were  most  frequently 
mentioned  during  presentations  and  discussions.  A  difficulty  in  summarizing  group  opinion 
on  most  important  cost  elements  such  as  these  results  from  the  difference  in  the  level  of 
detail  at  which  each  participant  presented  their  cost  factors.  Some  overlap  of  factors  is 
unavoidable;  for  example,  the  "amount  of  BIT"  is  a  detailed  factor  associated  with  a  more 
general  factor,  electronic  "equipment  type."  This  overlap  may  be  useful,  however,  given 
the  variability  in  the  amount  of  detail  for  information  available  on  a  given  system  at 
various  stages  in  development.  For  example,  one  may  only  know  that  a  certain 
percentage  of  equipment  will  be  electronic  and  not  the  amount  of  BIT  to  be  incorporated, 
at  the  time  a  3PA  cost  estimate  is  made. 

"Equipment  type"  includes  mechanical  and  electronic  systems.  Although  the  group 
agreed  that  mechanical  systems  outnumbered  electronic  ones  (for  the  Navy),  electronic 
systems  received  much  more  discussion  emphasis.  The  inclusion  of  troubleshooting  tasks 
was  determined  to  influence  complexity  and  cost  of  3PA  production  significantly. 
Although  BITE  and  ATE  were  offered  as  hardware  aids  for  troubleshooting  tasks, 
limitations  of  these  alternatives  showed  that  we  are  a  long  way  from  eliminating  a  man- 
in-the-loop.  Dr.  Inaba  indicated  that  a  65-  to  130-percent  page  increase  could  be 
expected  for  troubleshooting  tasks.  Improvement  of  front-end  analyses  and  computer- 
aided  maintenance  were  offered  as  troubleshooting  3PA  cost  reducers.  Hardware 
repetitions  and  symmetry  were  noted  as  electronic  equipment  characteristics  that  could 
be  capitalized  upon  in  reducing  3PA  preparation  costs. 
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Personnel  experience  level  or  prior  training  were  not  mentioned  as  factors,  although 
they  were  implied  in  the  "personnel  safety"  element.  The  unspoken  rules  covering 
user/data  matching  were  demonstrated  in  Mr.  Rahl's  examples  of  JPAs  for  various  user 
skill  levels.  It  was  mentioned  that  certain  graphics  are  usually  thought  of  as  being 
applicable  to  either  experienced  or  inexperienced  personnel;  however,  operational  ex¬ 
perience  had  not  shown  significant  performance  changes  when  the  rules  were  not 
followed.  Thus,  the  validity  question  concerning  commonly  used  user/data  matching  rules 
was  raised  but,  for  the  most  part,  avoided.  The  group  agreed  that  a  certain  format  could 
be  specified  as  "best"  for  a  given  user  group,  although  this  decision  was  not  accepted  as 
easy  to  do  in  practice. 

For  this  cost  area,  troubleshooting  tasks  received  the  most  discussion  as  a  cost 
element  and,  specifically,  troubleshooting  with  electronic  systems.  Clearly,  the  group 
identified  the  methodology  in  this  area  as  a  target  for  continued  research,  with  need  for 
improvements  in  guidelines  for  cost/effectiveness  tradeoffs  related  to  presentation 
methods  (especially  graphics). 

System  attributes,  such  as  number  of  LRUs,  SRUs,  subsystems,  etc.,  were  selected  as 
factors  within  page  estimation  algorithms.  These  attributes  may  offer  some  utility  as 
"easy  to  measure"  variables,  given  that  they  can  be  shown  to  be  associated  with  TI 
complexity/ volume. 

Procuring  Agency  Attributes 

Participants  who  addressed  "customer  characteristics"  as  a  driving  cost  factor  agreed 
that  this  factor  was  the  single  biggest  cost  inflator.  Contractors  all  had  stories  about 
lack  of  direction  and  personally  biased  tradeoff  decisions  emanating  from  systems 
acquisition  managers.  Participants  voiced  their  feelings  that  the  specifications  were 
good,  but  not  properly  implemented  due  to  the  influence  of  these  managers  and  the 
ignorance  of  procurers.  The  institutionalization  of  JPAs,  such  as  in  the  Army  SPA 
program,  was  viewed  as  a  first  step  in  reducing  the  negative  effect  of  these  project 
managers.  Education  of  acquisition  personnel  was  determined  to  be  critical.  Mr.  Joyce 
stated  that  a  responsible  position  has  to  be  implemented,  one  with  authority  and 
knowledge  to  implement  the  specifications  properly.  This  person  would  be  responsible  for 
obtaining  maintenance  performance  per  dollar  and  considering  the  effects  of  JPAs  on 
long-range  system  costs.  The  general  feeling  of  frustration  voiced  by  the  group  was  that 
we  could  have  the  best  specifications  in  the  world  concerning  JPA  production;  however, 
we  are  at  the  mercy  of  decision-makers  who  are  making  cost  tradeoff  decisions  while 
paying  more  attention  to  procurement  budgets  and  personal  feelings,  with  system 
effectiveness  and  specification  directions  considered  as  an  afterthought. 

To  develop  these  criteria,  the  methodology  for  quantifying  JPA  cost  impact  elements 
has  to  be  strengthened  to  provide  better  direction  for  cost/effectiveness  tradeoff 
decisions.  In  addition,  the  acquisition  manager  has  to  be  made  aware  of  the  results  of  this 
improved  JPA  methodology.  The  latter  is  necessary  if  the  first  is  to  have  any  impact. 

JPA  Producer  Attributes 


General  agreement  was  that  differences  among  producer-specific  cost-estimation 
methods  increases  the  difficulty  of  comparing  bidders'  proposals  by  a  procurement  agency. 
These  producers'  idiosyncrasies  do  not  significantly  affect  final  product  cost,  however. 
Mr.  Weber  concentrated  his  presentation  on  the  problems  of  developing  and  maintaining  a 
successful  JPA  production  staff.  Mr.  Bean  pointed  to  overhead  costs  as  a  significant 
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proportion  of  overall  costs.  These  cost  factors  were  generally  viewed  as  facts  of  life  or 
company-internal  factors.  In  relation  to  other  cost  factors,  it  seemed  that  the  majority 
of  attendees  would  downplay  the  relative  importance  of  company  peculiarities  in  JPA 
production  costs  with  procuring  agency  characteristics  and  front-end  analysis  problems 
far  outweighing  these  factors.  While  producer  factors  appear  to  have  some  cost  influence 
and  the  procurer  should  consider  each  bidders'  production-staff  experience,  these  factors 
don't  appear  to  be  a  prime  target  for  immediate  study/efforts  for  cost  reduction/predic- 
tion. 


JPA  Characteristics 


These  factors  include  the  physical  attributes  of  the  printed  JPA  materials.  As 
Figure  1  shows,  these  characteristics  are  (or  should  be)  defined  in  the  early  development 
phases  of  a  system.  The  type  of  format  chosen  is  generally  thought  to  be  dependent  upon 
task  variables,  user  skill  level,  and  working  environment.  Dr.  Smillie  noted  that  we  still 
don't  have  good  performance  data  to  support  these  format  tradeoff  decisions.  It  appears 
uncertain,  however,  how  much  cost  impact  these  JPA  characteristics  have  on  overall  cost. 
Mr.  Rahl  compared  the  overall  JPA  cost  to  an  iceberg  with  a  small  portion  visible  above 
the  water.  The  visible  portion  represented  the  attributes  of  the  final  product  that  were 
clearly  visible — number  of  pages,  density,  format,  illustrations,  color,  etc.  The  sub¬ 
merged  iceberg  portion  represented  other  factors  such  as  cost  areas  2,  5,  and  6  in  Table  1. 
Thus,  once  a  format  decision  is  made  and  the  input  data  collected,  the  cost  of  the  process 
of  turning  this  information  into  a  meaningful  format  varies  little  as  a  function  of  format 
chosen  in  relation  to  overall  cost  variability  caused  by  other  factors.  Production  costs  are 
becoming  increasingly  controlled  with  the  use  of  computer-aided  text  processing,  graph¬ 
ics,  and  authoring. 

These  cost  factors  will  inevitably  find  their  places  within  cost  estimation  models  but, 
as  this  conference  group  has  indicated,  the  models  with  a  sole  focus  on  product-attribute 
variables  will  generally  yield  ballpark  figures  and  the  majority  of  the  cost  variability  will 
be  uncontrolled  as  a  result  of  the  submerged  factors.  Group  consensus  was  as  follows: 
Given  two  sets  of  physically  similar  JPAs  produced  by  the  same  company,  you  could  likely 
have  a  large  cost  difference  due  to  the  variability  in  the  path  that  was  followed  from  the 
initial  JPA  requirement  to  the  final  product.  To  the  extent  that  cost  variability  for  the 
submerged  factors  can  be  reduced,  these  more  easily  measured  JPA  attribute  factors  will 
emerge  as  useful  cost  predictors. 

RFP/Bidding  Process 

The  first  factor  includes  the  preparation  of  the  RFP  and  the  JPA  producer  proposal 
response.  Vagueness  and  uncertainty  at  this  point  were  noted  by  all  contractors  as 
commonly  found  customer  attributes.  Quite  often  the  customer  isn't  sure  what  he  wants 
or  needs  but,  when  the  final  product  begins  to  become  a  physical  entity,  the  customer 
realizes  that  the  product  is  not  what  he  wants.  The  suggestion  that  a  work  sample  be 
included  in  the  RFP  would  certainly  help  reduce  this  uncertainty. 

It  was  apparent  that  the  majority  of  customers  do  not  know  how  to  utilize 
specifications  and  translate  their  ideas  into  written  JPA  requirements  during  early  phases 
in  the  development  process.  Efforts  such  as  that  described  by  Mr.  Finegan  are  aimed  at 
creating  better  guidelines  for  contract  monitors.  Clearly  specified  project  goals,  at  this 
point,  can  direct  effort  toward  a  final  product  with  minimal  wasted  energy/money. 
Uncertainty  was  identified  as  the  frequent  enemy  of  budget  directors,  procurers,  and 
producers  alike. 
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A  question  often  raised  during  the  meetings  was  the  location  of  responsibility  for 
accurate  cost  estimates  at  this  point  in  time.  At  this  point,  the  preparer  and  buyer  are 
trying  to  estimate  costs  from  the  viewpoint  of  how  much  money  is  available,  and  what 
technical  data  can  be  created  to  fill  the  void.  The  conference  participants  voiced  concern 
over  this  backwards  method  of  3PA  material  proposal  preparation,  but  could  offer  no 
positive  solutions  to  this  complex  problem.  Even  if  the  motivation  existed  in  all  parties 
involved  for  buying  a  certain  level  of  performance  with  TI  dollars,  current  state  of  art  in 
the  knowledge  of  cost-format-performance  relationships  prohibits  one  from  taking  that 
attitude. 

Other  suggestions  for  RFP  process  improvements  relevant  to  cost  control  included 
bidding  conferences,  freer  response  schedules,  and  separation  of  front-end  production 
costs.  The  impact  of  the  last  suggestion  on  overall  costs  depends  upon  the  type  and 
quantity  of  front-end  work  the  sponsor  expects  the  contractor  to  do  (or  redo).  This  must 
be  considered  in  the  RFP  statement. 

Mr.  Weber  explained  how  the  RFP  schedule  can  place  contractors  out  of  the  market 
when  interactions  of  schedule  and  certain  corporate  characteristics  occur.  The  bottom 
line  for  the  contractors,  at  this  point,  was  that  they'll  do  whatever  the  procurer  specifies 
in  the  RFP,  but  the  consequences  of  poor  input  data  and  changes  in  customer  direction 
will  greatly  increase  cost  or  result  in  a  poor  quality  product.  Project  direction  in  terms  of 
observable  goal,  work  samples,  knowledge  of  front-end  data  quality  and  requirements, 
user/data  match  guidelines  will  increase  the  accuracy  of  preproduction  cost  estimates, 
and  shift  the  cost  variability  to  the  production  process  phases. 

Input  Data  Quality 

The  quality  of  the  input  data,  the  process  of  collection,  and  the  availability  of  data 
sources  were  probably  the  most  talked  about  factors  during  the  conference.  Dr.  Inaba 
pointed  to  the  input  data  validity  as  one  of  two  major  cost  determinants.  He  felt  that  the 
responsibility  for  quality  input  data  lies  with  the  procurer  and  too  often  the  3PA  producer 
has  absorbed  the  responsibility  for  correcting  poor  quality  data.  Regardless  of  where  the 
responsibility  lies,  everyone  agreed  that  current  LSAs  are  not  providing  it  at  sufficient 
quality  levels.  Arguments  about  whether  the  problem  should  be  fixed  under  the  LSA 
umbrella  or  whether  a  separate  effort  was  needed  continued  for  some  time  during 
Mr.  Post's  presentation.  The  group  agreed  that  LSA  is  here  to  stay  and  that  something 
should  be  done  to  make  its  products  better  suited  for  technical  information  preparers.  It 
seems  that  much  of  this  deficient  quality  results  from  the  LSA  trying  to  be  everything  for 
everybody.  The  logistics  problem,  by  whom  and  when  the  data  are  collected,  was 
stressed. 

Improvement  suggestions  offered  were  as  follows:  Prepare  source  data  in  suitable 
format  for  3PA  producer  pick-up,  originate  data  from  disciplines  traditionally  assigned  to 
complete  it,  improve  LSA  in  Project  Hardman  context,  and  specify  equipment  availability 
as  part  of  the  program  plan.  The  group  consensus  was  that  someone  must  be  responsible 
for  these  data;  most  often  the  data  need  to  be  reconditioned.  The  procurer  must  be  aware 
of  the  need  for  quality  data  and  be  prepared  to  accept  the  responsibility  for  quality  or  be 
willing  to  spend  more  money  for  contractor  collection  of  this  data. 

Although  verification/validation  occurs  later  in  the  development  process,  they  are 
arbitrarily  included  in  the  "input  data"  factors  as  they  are  a  form  of  information  transfer 
from  the  system  to  the  producer.  Discussion  of  the  task  sample  size,  subject  availability, 
and  subject  sample  size  led  to  a  group  consensus  that:  (1)  Subjects  should  be  selected 
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from  the  target  population,  (2)  a  high  percentage  of  tasks  (at  least  95%)  should  be 
validated,  and  (3)  these  items  should  be  specified  in  the  contract.  Producers  were  nervous 
about  cutting  corners  during  validation/verification  in  an  effort  to  cut  costs.  Despite 
relatively  high  costs  and  difficulty,  the  accuracy  of  validations  and  generalization  of 
these  results  to  a  "real  world"  implementation  of  the  3PA  were  seen  as  too  important  to 
reputation  and  3PA  acceptance  to  risk  a  reduced  verification/validation  effort. 

3PA  Production  Process 


Contractors  generally  felt  that  they  had  control  over  cost  variability  for  the  3PA 
generation  process — writing,  illustrating,  editing,  etc. — and  computer  aiding  of  the  labor- 
intensive  areas  was  unanimously  acclaimed  to  be  the  precursor  to  tightened  cost  control. 
Some  overregulation  by  government  specifications  was  indicated  as  causing  too  much 
detail  in  text  and  overelaborate  illustrations.  Contractor-procurer  communication 
problems  during  this  process,  including  poor  start-up  meetings  and  lack  of  proper  in- 
process  reviews,  were  acknowledged. 

Production  costs  are  not  inexpensive.  Group  consensus  was  that  they  do  not 
significantly  contribute  to  differences  between  final  costs  and  initial  cost  estimates. 
Given  a  prescribed  set  of  front-end  conditions,  including  quality  input-data  and  format 
requirements,  the  procurer  can  be  reasonably  assured  that  the  cost  estimate  for 
production  will  be  accurate.  Separation  of  production  costs  and  up-front  costs  was  again 
voiced  as  a  practical  solution  to  many  cost  control  problems. 

3PA  Cost  Factors  Summary 

Table  1  also  addresses  major  contractor  characteristics  and  process  characteristics 
that  relate  to  cost  reduction.  In  addressing  cost  impact  of  processes,  the  procurer  must 
consider  who  will  bear  the  responsibility  for  each  development  facet  and  how  cost  will  be 
affected  by  tradeoff  decisions.  The  procurer  must  also  assess  the  "state"  of  each 
participant  in  the  development  process,  including  himself,  and  ask  how  deficiencies  will 
affect  cost.  Specific  questions  that  may  be  formulated  include  those  of  the  type:  Does 
the  contractor  have  computer-aided  processing?  Can  the  equipment  be  made  available 
locally  for  the  contractor?  Will  troubleshooting  tasks  be  involved?  Does  the  contractor 
have  an  experienced  3PA  production  team?  How  do  I  deal  with  poor  quality  input  data? 
How  much  will  it  cost  to  improve  the  quality  of  the  data? 

Although  dollar -amounts  cannot  be  fixed  to  the  answers  of  these  questions,  the 
general  magnitude  associated  with  many  tradeoff  choices  appears  to  be  estimable. 
Recognition  of  the  factors  involved  is  certainly  a  positive  first  step.  These  tradeoff 
factors  can  be  grouped  according  to: 

1.  Those  that  can  be  targeted  for  elimination  of  their  effects  on  cost  variability. 

2.  Those  that  can  be  quantitatively  measured  and  used  to  predict  production  or  up¬ 
front  costs. 

3.  Those  that  are  not  subject  to  elimination  of  quantitative  measurement  but  that 
can  be  qualitatively  assessed. 

Future  research  efforts  should  yield  results  that  are  useful  to  the  practitioner- -both 
procurer  and  preparer — and  they  should  focus  on  the  type  of  questions  indicated  above. 
Grouping  of  the  tradeoff  factors  into  the  three  types  noted  above  will  provide  some 
prioritization  for  these  efforts. 
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Recom  mendations 


1.  Identify  areas  where  BITE  and  ATE  may  be  harmful  to  the  troubleshooting 
technician  as  well  as  the  cost  tradeoffs  for  BITE  and  ATE. 

2.  Identify  areas  in  the  front-end  analysis  (e.g.,  the  failure  mode  effects  analysis 
and  dependency  analysis)  that  do  not  provide  adequate  troubleshooting  data  for  develop¬ 
ment  of  troubleshooting  JPAs. 

3.  Develop  guidelines  for  3PA  procurers  that  account  for  symmetry  in  electronic 
equipment. 

4.  Improve  cost/effectiveness  tradeoff  guidelines  for  the  user/data  match. 

5.  Improve  the  methodology  for  quantifying  the  JPA  cost  impact  elements. 

6.  Implement  an  education  program  for  acquisition  managers  and  procurement 
personnel  that  demonstrates  the  long-term  life  cycle  cost  effectiveness  of  the  JPA 
methodology. 

7.  Develop  criteria  to  reduce  the  variability  of  the  attributes  of  procuring  agencies. 

8.  Develop  strategy  that  places  burden  on  the  contractor  for  an  accurate  cost 
estimate. 

9.  Develop  guidelines  for  including  work  samples  in  procurement  packages. 

10.  Improve  state-of-the-art  methods  for  defining  and  quantifying  cost-format- 
performance  relationships. 

11.  Develop  guidelines  for  establishing  a  bidders'  conference  that  can  be  used  to 
increase  the  understanding  of  requirements  and  clarify  customer  uncertainty. 

12.  Develop  guidelines  that  adequately  reflect  the  separation  of  front-end  costs 
from  JPA  production  costs. 

13.  Develop  guidelines  that  delineate  the  responsibiities  for  quality  input  data  for 
the  JPA  development  process. 

14.  Determine  the  relationship  of  LSA  to  the  JPA  development  process  and  identify 
the  JPA  input  data  gaps. 

15.  Establish  cost-effective  guidelines  for  JPA  validation/ verification  that  address 
task  sample  size,  user  sample  size,  and  availability  of  equipment. 

16.  Identify  specific  areas  where  production  specifications  are  forcing  increased 
volume  and  cost,  and  develop  guidelines  for  improving  the  process. 
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LIST  OF  ABBREVIATIONS  AND  ACRONYMS 


ATE 

BIT 

BITE 

C  ATI  PS 

DARCOM 

EPICS 

FMEA 

FP3PA 

ILS 

I  PR 

3PA 

LCC 

LRU 

LSA 

LSAR 

MAC 

MDC 

MRC 

NAVPERSRANDCEN 

NSWSES 

NTIPP 

NTIPS 

PAGES 

PIMO 

POC 

Q/A 

RFP 

RFQ 

SME 

SPA 

SPO 

SRU 

TI 

TIM 

TM 

TRADOC 


Automated  test  equipment 

Built-in  test 

Built-in  test  equipment 

Computer-aided  technical  information  preparation  system 

(Army)  Material  Readiness  Command 

Enlisted  Personnel  Individualized  Career  System 

Failure  mode  effect  analysis 

Fully  proceduralized  job  performance  aid 

Integrated  logistics  support 

In  process  review 

3ob  performance  aid 

Life  cycle  cost 

Line  replaceable  unit 

Logistic  support  analysis 

Logistic  support  analysis  record 

Maintenance  allocation  chart 

Maintenance  dependency  chart 

Maintenance  requirement  card 

Navy  Personnel  Research  and  Development  Center 

Naval  Ship  Weapon  Systems  Engineering  Station 

Navy  technical  information  presentation  program 

Navy  technical  information  presentation  system 

Computer-based  technical  order  assignment  method 

Presentation  of  information  for  maintenance  organization 

Point  of  contact 

Quality  assurance 

Request  for  proposal 

Request  for  quotation 

Subject  matter  expert 

Skilled  performance  aid 

System  program  office 

Shop  replaceable  unit 

Technical  information 

Task  identification  matrix 

Technical  manual 

(Army)  Training  and  Doctrine  Command 
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APPENDIX 

POST-CONFERENCE  COMMENTS 
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POST-CONFERENCE  COMMENTS 


At  the  conclusion  of  the  conference,  participants  were  asked  to  identify  the  one  JPA 
cost  topic  that  they  felt  was  either  the  most  important  or  needed  immediate  attention. 
Four  participants  volunteered  the  following  comments. 

Reid  P.  Joyce 

It  seemed  to  me  that  much  of  the  discussion  dealt  with  the  nuts  and  bolts  of  making 
JPAs  and  that  we  might  have  been  able  to  address  a  number  of  issues  that  face  the  buyer, 
if  we'd  had  a  bit  more  time  beyond  that  allocated  to  "presentations.”  It  strikes  me  that 
what  we  were  hearing  was  that  several  competent  organizations  have  different- -but 
maybe  equally  good — ways  to  build  JPAs,  and  some  of  these  differences  have  cost 
implications.  The  questions  that  I  think  we  didn't  deal  with  adequately  though  can  have 
cost  leverage  implications  that  far  outweigh  the  intercompany  differences  in  JPA-building 
cost: 


•  Who  does  the  front-end  analysis? 

•  How  should  contractors  and  buyers  deal  with  uncertainty  in  quality  and  quantity 
of  front-end  analysis  data? 

•  If  it's  true  that  you  can't  closely  estimate  JPA  cost  until  some  kind  of  front-end 
analysis  is  done,  how  can  you  select  a  JPA  contractor? 

•  If  most  JPA  contractors  can  control  quality  pretty  well,  how  do  you  choose  the 
one  who  will  ultimately  give  you  the  most  maintenance  performance  per  dollar? 

•  Can  we  ever  have  technical  data  acquisition  managers  who  are  permitted  to  care 
about  maintenance  performance  per  dollar  and  long-range  cost  of  system  ownership? 

•  Who  would  train  such  an  acquisition  manager,  if  the  job  included  enough 
authority  to  apply  the  kind  of  wisdom  he  ought  to  have? 

I  have  the  feeling  that  all  too  often  we  "back  into"  a  JPA-scoping  effort  by  trying  to 
meet  an  arbitrarily  established  budget,  rather  than  truly  fighting  for  a  share  of  the 
system-development  dollars  that's  commensurate  with  the  JPAs'  contribution  to  life  cycle 
cost  of  the  system.  Until  we  can  make  that  argument  convincingly,  we  won't  get  the 
bucks,  and  we'll  be  chronically  cutting  corners  and  making  compromises  that  we  shouldn't 
have  to  make. 

Fred  L.  Hart 


In  my  opinion,  the  point  brought  out  most  often  was  that  the  procurement  activity 
must  be  more  knowledgeable  in  what  options  are  available  in  the  purchase  of  technical 
documentation.  These  options  are  available  in  the  purchase  of  technical  documentation. 
These  options  are  in  terms  of  the  types  and  levels  of  presentation  that  are  available  and 
that  may  be  procured  to  a  set  specification.  The  decisions  made  by  the  procuring  activity 
must  be  made  with  full  knowledge  of  the  maintenance  philosophy  of  the  equipment/system 
and  the  level  of  personnel  expected  to  maintain  the  equipment  (both  entry  level  and 
experienced  personnel).  The  use  and  impact  of  this  documentation  on  training  must  also 
be  considered.  Too  often  training  tradeoffs  are  made  without  regard  to  the  potential  use 
of  the  equipment/system  technical  documentation.  A  corollary  to  the  above  is  that  the 
procurement  activity  must  also  take  the  time  to  consider  the  life  cycle  cost  impacts  of 
technical  documentation. 
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Rosemarie  3.  Preidis 


I  wish  we  could  have  arrived  at  a  consensus  on  the  most  influential  factors  that 
influence  cost  variability.  We  discussed  many  factors  (e.g.,  design  stability,  customer 
direction,  illustration  and  information  density,  task  analysis).  We  should  have  inspected 
those  factors  in  more  detail  to  separate  the  high  impact  ones  from  the  mediocre.  We  also 
should  have  delved  more  deeply  into  figuring  out  how  to  capture  them  quantitatively. 

I  think  we  established  a  good  knowledge  base  from  our  collective  inputs.  However,  it 
will  need  more  massaging  before  definitive  cost-estimation  guidelines  can  be  formulated. 
Hopefully  your  workshop  proceedings  overview  will  shed  some  light  on  the  direction  we 
should  take  to  develop  useful  guidelines. 

3ohn  G.  Bean 


I  recommend  development  of  a  simple  and  usable  cost  estimating  model.  This 
requires  two  related  tasks:  first,  algorithms  to  estimate  the  number  of  pages  for  various 
types  of  documents  using  various  factors  such  as  number  of  systems,  units,  modules,  etc. 
(similar  to  the  model  developed  by  Rosemarie  Preidis);  and  second,  cost  per  page 
standards  for  each  type  of  document.  A  very  few  types  of  documents  could  be  considered 
at  first.  A  limited  number  of  cost  per  page  categories  should  be  established  for  each 
type. 


Document  Type 

1.  Fully  proceduralized  3PA 

2.  Hybrid  3PA 
3.3PA 

4.  Job  guide 

5.  Maintenance 

6.  Flight 

7.  Operator 

8.  Illustrated  parts  breakdown 

9.  Overhaul 

10.  Checklist 


Cost-per-page  Category 


1. 

Writing 

(Hr/Pg) 

2. 

Illustrating 

(Hr/Pg) 

3. 

T  yping/ production 

(Hr/Pg) 

4. 

Inspection 

(Hr/Pg) 

5. 

Validation 

(Hr/Pg) 

6. 

Material 

($/Pg) 

It  would  be  necessary  to  furnish  a  sample  of  each  type  of  document,  so  it  would  be 
clear  what  is  specified  (page  size,  type  size,  single  column/double  column,  etc.).  To 
obtain  accurate  historical  data,  each  contractor  must  establish  a  fairly  elaborate 
accounting  system  in  the  cost-per-page  categories  in  which  to  collect  costs  for  each 
document  type.  It  is  possible  to  do  this  for  a  simple  system.  For  a  system  such  as  the  one 
described  by  Rosemarie,  the  necessary  cost  accounting  system  would  be  so  elaborate  and 
its  costs  so  excessive  that  it  does  not  appear  feasible  to  me  to  collect  so  much  detail. 

Another  important  area  for  which  work  might  continue  is  the  methodology  of  making 
tradeoffs  between  JPAs  and  training.  These  tradeoffs  are  usually  not  considered  during 
the  logistics  system  design  phase,  yet,  as  EPICS  has  shown,  there  are  considerable 
implications  of  being  able  to  make  the  tradeoff  studies  effectively.  The  standards 
discussed  above  could  be  developed  over  a  range  of  document  types  (FPJPA,  JPA, 
conventional,  commercial,  data  package,  design  package)  to  enable  the  necessary  cost 
estimates  to  be  made. 
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